OpenModelZ
One-click machine learning deployment at scale on any cluster (GCP, AWS, Lambda labs, your home lab, or even a single machine).
Why use OpenModelZ
OpenModelZ is the ideal solution for practitioners who want to quickly deploy their machine learning models to a (public or private) endpoint without the hassle of spending excessive time, money, and effort to figure out the entire end-to-end process.
We created OpenModelZ in response to the difficulties of finding a simple, cost-effective way to get models into production fast. Traditional deployment methods can be complex and time-consuming, requiring significant effort and resources to get models up and running.
- Kubernetes: Setting up and maintaining Kubernetes and Kubeflow can be challenging due to their technical complexity. Data scientists spend significant time configuring and debugging infrastructure instead of focusing on model development.
- Managed services: Alternatively, using a managed service like AWS SageMaker can be expensive and inflexible, limiting the ability to customize deployment options.
- Virtual machines: As an alternative, setting up a cloud VM-based solution requires learning complex infrastructure concepts like load balancers, ingress controllers, and other components. This takes a lot of specialized knowledge and resources.
With OpenModelZ, we take care of the underlying technical details for you, and provide a simple and easy-to-use CLI to deploy your models to any cloud (GCP, AWS, or others), your home lab, or even a single machine.
You could start from a single machine and scale it up to a cluster of machines without any hassle. Besides this, We provision a separate subdomain for each deployment without any extra cost and effort, making each deployment easily accessible from the outside.
OpenModelZ forms the core of our ModelZ (opens in a new tab) platform, which is a serverless machine learning inference service. It is utilized in a production environment to provision models for our clients.
Quick Start ๐
Install mdz
You can install OpenModelZ using the following command:
pip install openmodelz
You could verify the installation by running the following command:
mdz
Once you've installed the mdz
you can start deploying models and experimenting with them.
Bootstrap mdz
It's super easy to bootstrap the mdz
server. You just need to find a server (could be a cloud VM, a home lab, or even a single machine) and run the mdz server start
command. The mdz
server will be bootstrapped on the server as a controller node and you could start deploying your models.
We will require the root permission to bootstrap the mdz
server in port 80.
$ mdz server start
๐ง Creating the server...
๐ง Initializing the load balancer...
๐ง Initializing the GPU resource...
๐ง Initializing the server...
๐ง Waiting for the server to be ready...
๐ Checking if the server is running...
Agent:
Version: v0.0.13
Build Date: 2023-07-19T09:12:55Z
Git Commit: 84d0171640453e9272f78a63e621392e93ef6bbb
Git State: clean
Go Version: go1.19.10
Compiler: gc
Platform: linux/amd64
๐ณ The server is running at http://192.168.71.93.modelz.live
๐ You could set the environment variable to get started!
export MDZ_URL=http://192.168.71.93.modelz.live
The internal IP address will be used as the default endpoint of your deployments. You could provide the public IP address of your server to the mdz server start
command to make it accessible from the outside world.
# Provide the public IP as an argument
$ mdz server start 1.2.3.4
You could also specify the registry mirror to speed up the image pulling process. Here is an example:
$ mdz server start --mirror-endpoints https://docker.mirrors.sjtug.sjtu.edu.cn
Create your first UI-based deployment
Once you've bootstrapped the mdz
server, you can start deploying your first applications. We will use jupyter notebook as an example in this tutorial. You could use any docker image as your deployment.
$ mdz deploy --image jupyter/minimal-notebook:lab-4.0.3 --name jupyter --port 8888 --command "jupyter notebook --ip='*' --NotebookApp.token='' --NotebookApp.password=''"
Inference jupyter is created
$ mdz list
NAME ENDPOINT STATUS INVOCATIONS REPLICAS
jupyter http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live Ready 488 1/1
http://192.168.71.93/inference/jupyter.default
You could access the deployment by visiting the endpoint URL. The endpoint will be automatically generated for each deployment with the following format: <name>-<random-string>.<ip>.modelz.live
.
It is http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live
in this case. The endpoint could be accessed from the outside world as well if you've provided the public IP address of your server to the mdz server start
command.
Create your first OpenAI compatible API server
You could also create API-based deployments. We will use OpenAI compatible API server with Bloomz 560M (opens in a new tab) as an example in this tutorial.
$ mdz deploy --image modelzai/llm-bloomz-560m:23.07.4 --name simple-server
Inference simple-server is created
$ mdz list
NAME ENDPOINT STATUS INVOCATIONS REPLICAS
jupyter http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live Ready 488 1/1
http://192.168.71.93/inference/jupyter.default
simple-server http://simple-server-lagn8m9m8648q6kx.192.168.71.93.modelz.live Ready 0 1/1
http://192.168.71.93/inference/simple-server.default
You could use OpenAI python package and the endpoint http://simple-server-lagn8m9m8648q6kx.192.168.71.93.modelz.live
in this case, to interact with the deployment.
import openai
openai.api_base="http://simple-server-lagn8m9m8648q6kx.192.168.71.93.modelz.live"
openai.api_key="any"
# create a chat completion
chat_completion = openai.ChatCompletion.create(model="bloomz", messages=[
{"role": "user", "content": "Who are you?"},
{"role": "assistant", "content": "I am a student"},
{"role": "user", "content": "What do you learn?"},
], max_tokens=100)
Scale your deployment
You could scale your deployment by using the mdz scale
command.
$ mdz scale simple-server --replicas 3
The requests will be load balanced between the replicas of your deployment.
You could also tell the mdz
to autoscale your deployment based on the inflight requests. Please check out the Autoscaling section for more details.
Debug your deployment
Sometimes you may want to debug your deployment. You could use the mdz logs
command to get the logs of your deployment.
$ mdz logs simple-server
simple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:16] "GET / HTTP/1.1" 200 -
simple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:16] "GET / HTTP/1.1" 200 -
simple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:17] "GET / HTTP/1.1" 200 -
You could also use the mdz exec
command to execute a command in the container of your deployment. You do not need to ssh into the server to do that.
$ mdz exec simple-server ps
PID USER TIME COMMAND
1 root 0:00 /usr/bin/dumb-init /bin/sh -c python3 -m http.server 80
7 root 0:00 /bin/sh -c python3 -m http.server 80
8 root 0:00 python3 -m http.server 80
9 root 0:00 ps
$ mdz exec simple-server -ti bash
bash-4.4#
Or you could port-forward the deployment to your local machine and debug it locally.
$ mdz port-forward simple-server 7860
Forwarding inference simple-server to local port 7860
Add more servers
You could add more servers to your cluster by using the mdz server join
command. The mdz
server will be bootstrapped on the server and join the cluster automatically.
$ mdz server join <internal ip address of the previous server>
$ mdz server list
NAME PHASE ALLOCATABLE CAPACITY
node1 Ready cpu: 16 cpu: 16
mem: 32784748Ki mem: 32784748Ki
gpu: 1 gpu: 1
node2 Ready cpu: 16 cpu: 16
mem: 32784748Ki mem: 32784748Ki
gpu: 1 gpu: 1
Label your servers
You could label your servers to deploy your models to specific servers. For example, you could label your servers with gpu=true
and deploy your models to servers with GPUs.
$ mdz server label node1 gpu=true type=nvidia-a100
$ mdz deploy --image modelzai/llm-bloomz-560m:23.07.4 --name simple-server --node-labels gpu=true,type=nvidia-a100
If you have different type of GPUs, you can label your nodes with mdz server label node3 gpu=true type=nvidia-a100
and deploy the serevice to the selected nodes with mdz deploy --image llama2 --node-labels gpu=true,type=nvidia-a100
.