mlc-chat - RedPajama-INCITE-Chat-3B on macOS#

MLC (Machine Learning Compilation) on May 22nd 2023: Bringing Open Large Language Models to Consumer Devices

RedPajama on Apple Silicon is achieved by compiling the LLM using Metal for M1/M2 GPUs

The instructions they provided didn’t quite give me all the information I needed to get this to work. Here’s what I did instead.

UPDATE: The official instructions now cover RedPajama-INCITE-Chat-3B as well.

Install Conda#

Their instructions require Conda. I installed miniconda using Homebrew like this:

1
brew install --cask miniconda

Following their initial instructions for Vicuna-7B#

These worked for getting the mlc-chat-vicuna-v1-7b model running on my laptop.

First I created mlc-chat/dist/prebuilt and pulled some models and files from their repos:

1
mkdir mlc-chat
2
cd mlc-chat
3
mkdir -p dist/prebuilt
4
# git lfs install - not necessary, I already had Git LFS
5
cd dist/prebuilt
6
git clone https://huggingface.co/mlc-ai/mlc-chat-vicuna-v1-7b-q3f16_0
7
git clone https://github.com/mlc-ai/binary-mlc-llm-libs.git lib
8
cd ../..

Then I created a Conda environment and installed some packages into that:

1
conda create -n mlc-chat
2
conda activate mlc-chat
3
conda install -c mlc-ai -c conda-forge mlc-chat-nightly
4
conda init zsh
5
# Quit and re-open terminal
6
cd mlc-chat
7
conda activate mlc-chat

Having done this, I could run the Vicuna model like so:

1
mlc_chat_cli --local-id vicuna-v1-7b-q3f16_0

1
Use MLC config: "/Users/simon/not-in-backups/mlc-chat/dist/prebuilt/mlc-chat-vicuna-v1-7b-q3f16_0/mlc-chat-config.json"
2
Use model weights: "/Users/simon/not-in-backups/mlc-chat/dist/prebuilt/mlc-chat-vicuna-v1-7b-q3f16_0/ndarray-cache.json"
3
Use model library: "/Users/simon/not-in-backups/mlc-chat/dist/prebuilt/lib/vicuna-v1-7b-q3f16_0-metal.so"
4
Loading model...
5
You can use the following special commands:
6
  /help               print the special commands
7
  /exit               quit the cli
8
  /stats              print out the latest stats (token/sec)
9
  /reset              restart a fresh chat
10
  /reload [local_id]  reload model "local_id" from disk, or reload the current model if `local_id` is not specified
11

12
[11:59:01] /Users/catalyst/Workspace/mlc-chat-conda-build/tvm/src/runtime/metal/metal_device_api.mm:165: Intializing Metal device 0, name=Apple M2 Max
13
USER: Write a rap battle between a pelican and a cheesecake
14
ASSISTANT: (Verse 1)
15
I'm a pelican, I'm the king of the sea
16
I'm a big bird with a big appetite for cheesecake
17
I'll be taking that piece of cheesecake right off your plate
18
I'll be feasting on the creamy goodness that's sweet
19

20
(Chorus)
21
You'll never see me coming for that slice
22
I'll be the one taking it all in this lunch time
23
So beware my sharp beak and talons, I'm coming for that slice
24
I'll be the one with a big appetite for that cheesecake good
25
# ...

Figuring out how to run RedPajama-INCITE-Chat-3B#

This wasn’t covered in their documentation, but I figured it out through trial and error.

1
cd dist/prebuilt
2
git clone https://huggingface.co/mlc-ai/mlc-chat-RedPajama-INCITE-Chat-3B-v1-q4f16_0
3
cd ../..

Now I can run that model like so:

1
mlc_chat_cli --local-id RedPajama-INCITE-Chat-3B-v1-q4f16_0

1
Use MLC config: "/Users/simon/not-in-backups/mlc-chat/dist/prebuilt/mlc-chat-RedPajama-INCITE-Chat-3B-v1-q4f16_0/mlc-chat-config.json"
2
Use model weights: "/Users/simon/not-in-backups/mlc-chat/dist/prebuilt/mlc-chat-RedPajama-INCITE-Chat-3B-v1-q4f16_0/ndarray-cache.json"
3
Use model library: "/Users/simon/not-in-backups/mlc-chat/dist/prebuilt/lib/RedPajama-INCITE-Chat-3B-v1-q4f16_0-metal.so"
4
Loading model...
5
You can use the following special commands:
6
  /help               print the special commands
7
  /exit               quit the cli
8
  /stats              print out the latest stats (token/sec)
9
  /reset              restart a fresh chat
10
  /reload [local_id]  reload model "local_id" from disk, or reload the current model if `local_id` is not specified
11

12
[12:01:41] /Users/catalyst/Workspace/mlc-chat-conda-build/tvm/src/runtime/metal/metal_device_api.mm:165: Intializing Metal device 0, name=Apple M2 Max
13
<human>: Write a rap battle between a pelican and a cheesecake
14
<bot>: Who's got more bounce?
15
The pelican's got a bounce like no other.
16

17
Pelicans are known for their unique style,
18
they're like a flying dunk in the truest form.
19
They're like a wall in the sky,
20
a wall that can't be broken.
21

22
But I'd argue that I've got more bounce
23
than you could ever imagine,
24
I'm like a cheesecake with a lot more flavor,
25
I'm like a cheesecake that'll never quit.
26

27
I'm like a cheesecake with a lot more flavor,
28
I'm like a cheesecake that'll never quit,
29
I'm like a cheesecake that'll never quit.
30

31
I'm like a cheesecake that'll never quit,
32
I'm like a cheesecake that'll never quit,
33
I'm like a cheesecake that'll never quit.