A blueprint for designing production LLM systems: From Notebooks to production

How to get a GitHub Copilot subscription for FREE (to 5x writing code). Learn to build production ML systems by building an LLM application.

Decoding ML Notes

This week’s topics:

• How to get a GitHub Copilot subscription for FREE (to 5x writing code)

• A blueprint for designing production LLM systems: From Notebooks to production

• Learn to build production ML systems by building an LLM application

---

How to get a GitHub Copilot subscription for FREE (to 5x writing code)

𝗛𝗼𝘄 to get a 𝗚𝗶𝘁𝗛𝘂𝗯 𝗖𝗼𝗽𝗶𝗹𝗼𝘁 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗽𝘁𝗶𝗼𝗻 for 𝗙𝗥𝗘𝗘 (to 5x writing code) ↓

There are other alternatives, but GitHub Copilot is still the leading solution due to 2 factors: performance & convenience.

If you can get it for free, there are 0 reasons not to use it (sneaky move Microsoft) ↓

𝗦𝗼 𝘄𝗵𝗮𝘁 𝗶𝘀 𝘁𝗵𝗲 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻?

There is no secret.

As stated in their docs: "Verified students, teachers, and maintainers of popular open source projects on GitHub are eligible to use Copilot Individual for free. "

🔗 Docs

To become a student or teacher when you are not is not a solution.

But...

To become a maintainer of a popular open-source project is!

𝗦𝗼 𝘄𝗵𝗮𝘁 𝗮𝗿𝗲 𝘁𝗵𝗲 𝗰𝗿𝗶𝘁𝗲𝗿𝗶𝗮 𝗳𝗼𝗿 𝗯𝗲𝗰𝗼𝗺𝗶𝗻𝗴 𝗮 "𝗺𝗮𝗶𝗻𝘁𝗮𝗶𝗻𝗲𝗿 𝗼𝗳 𝗮 𝗽𝗼𝗽𝘂𝗹𝗮𝗿 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 𝗽𝗿𝗼𝗷𝗲𝗰𝘁"?

I don't know the exact formula, but here are some examples.

I am eligible for it because I am the owner of a GitHub repository with ~2.2k stars & 350 forks: 🔗 Hands-on LLMs Course

After digging into some Reddit threads, a dude said that for a repo with ~520 stars & 299 forks, you got the free subscription.

The idea is that you don't have to be a maintainer of Pandas or PyTorch to become eligible.

.

𝗧𝗵𝗲 𝗰𝗼𝗻𝗰𝗹𝘂𝘀𝗶𝗼𝗻 𝗶𝘀 𝘁𝗼...

→ start contributing to open-source or creating your cool project, which will complete the job!

.

𝘐𝘧 𝘺𝘰𝘶 𝘣𝘦𝘵𝘵𝘦𝘳 𝘬𝘯𝘰𝘸 𝘵𝘩𝘦 "𝘴𝘦𝘤𝘳𝘦𝘵 𝘧𝘰𝘳𝘮𝘶𝘭𝘢/𝘤𝘳𝘪𝘵𝘦𝘳𝘪𝘢," 𝘱𝘭𝘦𝘢𝘴𝘦 𝘭𝘦𝘢𝘷𝘦 𝘪𝘵 𝘪𝘯 𝘵𝘩𝘦 𝘤𝘰𝘮𝘮𝘦𝘯𝘵𝘴 𝘧𝘰𝘳 𝘰𝘵𝘩𝘦𝘳𝘴 𝘵𝘰 𝘬𝘯𝘰𝘸.

Also, let me know if you know that when contributing to open-source, you must contribute by "how much" until you become eligible.

---

A blueprint for designing production LLM systems: From Notebooks to production

I am 𝗾𝘂𝗶𝘁𝘁𝗶𝗻𝗴 𝗰𝗿𝗲𝗮𝘁𝗶𝗻𝗴 𝗰𝗼𝗻𝘁𝗲𝗻𝘁... 𝗝𝗼𝗸𝗶𝗻𝗴, but here is 𝗵𝗼𝘄 to 𝗯𝘂𝗶𝗹𝗱 your 𝗟𝗟𝗠 𝘁𝘄𝗶𝗻 for 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗻𝗴 posts or articles 𝘂𝘀𝗶𝗻𝗴 𝘆𝗼𝘂𝗿 𝘃𝗼𝗶𝗰𝗲 ↓

𝗪𝗵𝗮𝘁 𝗶𝘀 𝗮𝗻 𝗟𝗟𝗠 𝘁𝘄𝗶𝗻?

It's an AI character who writes like you, using your writing style and personality.

𝗪𝗵𝘆 𝗻𝗼𝘁 𝗱𝗶𝗿𝗲𝗰𝘁𝗹𝘆 𝘂𝘀𝗲 𝗖𝗵𝗮𝘁𝗚𝗣𝗧? 𝗬𝗼𝘂 𝗺𝗮𝘆 𝗮𝘀𝗸...

When generating content using an LLM, the results tend to:

- be very generic and unarticulated,
- contain misinformation (due to hallucination),
- require tedious prompting to achieve the desired result.

𝗧𝗵𝗮𝘁 𝗶𝘀 𝘄𝗵𝘆, 𝗳𝗼𝗿 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗻𝗴 𝗰𝗼𝗻𝘁𝗲𝗻𝘁, 𝘆𝗼𝘂 𝗻𝗲𝗲𝗱 𝗮 𝘀𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘇𝗲𝗱 𝘁𝗼𝗼𝗹 𝘁𝗵𝗮𝘁:

→ is fine-tuned on your digital content to replicate your persona

→ has access to a vector DB (with relevant data) to avoid hallucinating and write only about concrete facts

𝗛𝗲𝗿𝗲 𝗮𝗿𝗲 𝘁𝗵𝗲 𝗺𝗮𝗶𝗻 𝘀𝘁𝗲𝗽𝘀 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝗱 𝘁𝗼 𝗯𝘂𝗶𝗹𝗱 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗿𝗲𝗮𝗱𝘆 𝗟𝗟𝗠 𝘁𝘄𝗶𝗻:

1. A data collection pipeline will gather your digital data from Medium, Substack, LinkedIn and GitHub. It will be normalized and saved to a Mongo DB.

2. Using CDC, you listen to any changes made to the Mongo DB and add them as events to a RabbitMQ queue.

3. A Bytewax streaming ingestion pipeline will listen to the queue to clean, chunk, and embed the data in real time.

4. The cleaned and embedded data is loaded to a Qdrant vector DB.

5. On the training pipeline side, you use a vector DB retrieval client to build your training dataset, which consists of the cleaned data (augmented using RAG).

6. You fine-tune an open-source Mistral LLM using QLoRA and push all the experiment artifacts to a Comet experiment tracker.

7. Based on the best experiment, you push the LLM candidate to Comet's model registry. You carefully evaluate the LLM candidate using Comet's prompt monitoring dashboard. If the evaluation passes, you tag it as accepted.

8. On the inference pipeline side, you deploy the new LLM model by pulling it from the model registry, loading it, and quantizing it.

9. The inference pipeline is wrapped by a REST API, which allows users to make ChatGPT-like requests.

---

Learn to build production ML systems by building an LLM application

Taking in mind the blueprint for designing production LLM systems presented above, we want to let you know that:

→ We are close to wrapping our LLM twin course lessons and code.

To give more context for newcomers, in the past weeks we started 𝗿𝗲𝗹𝗲𝗮𝘀𝗶𝗻𝗴 an 𝗲𝗻𝗱-𝘁𝗼-𝗲𝗻𝗱 𝗰𝗼𝘂𝗿𝘀𝗲 on 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗟𝗟𝗠𝘀 by teaching you how to 𝗯𝘂𝗶𝗹𝗱 an 𝗟𝗟𝗠 𝘁𝘄𝗶𝗻: 𝘠𝘰𝘶𝘳 𝘗𝘳𝘰𝘥𝘶𝘤𝘵𝘪𝘰𝘯-𝘙𝘦𝘢𝘥𝘺 𝘈𝘐 𝘙𝘦𝘱𝘭𝘪𝘤𝘢

So…

If you are looking for an 𝗲𝗻𝗱-𝘁𝗼-𝗲𝗻𝗱 𝗙𝗥𝗘𝗘 𝗰𝗼𝘂𝗿𝘀𝗲 on 𝗵𝗼𝘄 to 𝗯𝘂𝗶𝗹𝗱 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗿𝗲𝗮𝗱𝘆 𝗟𝗟𝗠 𝘀𝘆𝘀𝘁𝗲𝗺𝘀, consider checking the course's first FREE lesson.

𝘛𝘩𝘦 𝘤𝘰𝘶𝘳𝘴𝘦 𝘸𝘪𝘭𝘭 𝘸𝘢𝘭𝘬 𝘺𝘰𝘶 𝘵𝘩𝘳𝘰𝘶𝘨𝘩 𝘢 𝘧𝘶𝘭𝘭-𝘴𝘵𝘢𝘤𝘬 𝘱𝘳𝘰𝘤𝘦𝘴𝘴:

→ from data gathering...

...until deploying and monitoring your LLM twin using LLMOps ←

.

With that in mind...

The 𝟭𝘀𝘁 𝗹𝗲𝘀𝘀𝗼𝗻 will walk you through:

- the issues of generating content using ChatGPT (or other similar solutions)
- the 3-pipeline design
- the system design and architecture of the LLM twin

.

Within the 𝘀𝘆𝘀𝘁𝗲𝗺 𝗱𝗲𝘀𝗶𝗴𝗻 𝘀𝗲𝗰𝘁𝗶𝗼𝗻, we will present all the 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗮𝗹 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻𝘀 on 𝗵𝗼𝘄 to 𝗯𝘂𝗶𝗹𝗱:

- a data collection pipeline
- a real-time feature pipeline using a streaming engine
- hook the data and feature pipelines using the CDC pattern
- a continuous fine-tuning pipeline
- an inference pipeline deployed as a REST API


A 𝗽𝗮𝗿𝘁𝗶𝗰𝘂𝗹𝗮𝗿 𝗳𝗼𝗰𝘂𝘀 will be on 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗻𝗴 𝗠𝗟𝗢𝗽𝘀 & 𝗟𝗟𝗠𝗢𝗽𝘀 𝗴𝗼𝗼𝗱 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀:

- prompt versioning
- model registries
- experiment tracker
- prompt monitoring
- CI/CD
- IaC
- Docker

.

𝙒𝙖𝙣𝙩 𝙩𝙤 𝙙𝙞𝙜 𝙞𝙣𝙩𝙤 𝙩𝙝𝙚 1𝙨𝙩 𝙡𝙚𝙨𝙨𝙤𝙣?

𝗖𝗵𝗲𝗰𝗸 𝗶𝘁 𝗼𝘂𝘁. It's FREE, and no registration is required

↓↓↓

🔗 𝘓𝘦𝘴𝘴𝘰𝘯 1 - 𝘈𝘯 𝘌𝘯𝘥-𝘵𝘰-𝘌𝘯𝘥 𝘍𝘳𝘢𝘮𝘦𝘸𝘰𝘳𝘬 𝘧𝘰𝘳 𝘗𝘳𝘰𝘥𝘶𝘤𝘵𝘪𝘰𝘯-𝘙𝘦𝘢𝘥𝘺 𝘓𝘓𝘔 𝘚𝘺𝘴𝘵𝘦𝘮𝘴 𝘣𝘺 𝘉𝘶𝘪𝘭𝘥𝘪𝘯𝘨 𝘠𝘰𝘶𝘳 𝘓𝘓𝘔 𝘛𝘸𝘪𝘯

---

Images

If not otherwise stated, all images are created by the author.