Scale is proud to be a <a href="https://ai.meta.com/blog/meta-llama-3-1/" target="_blank" rel="noopener">Llama 3.1 Launch Partner</a>! Llama 3.1 405B is the largest openly available foundation model with capabilities that rival the best closed-source models. <a href="https://scale.com/generative-ai-data-engine">Scale Data Engine</a> provided a large amount of data for supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) to help make Llama 3.1 the leading open-source LLM. Now, Meta and Scale also partnered to help businesses customize, evaluate, and deploy Llama 3.1 405B for enterprise use cases using <a href="https://scale.com/genai-platform">Scale GenAI Platform</a>.
Only 10% of enterprises have GenAI in production, caused by a lack of trust in the performance and reliability of models. For enterprises to bridge this trust gap, they need to systematically <a href="https://scale.com/evaluation/enterprise">evaluate, improve, and monitor GenAI systems</a> for performance, safety, and reliability. Unbiased evaluations of LLMs for enterprise tasks are difficult, as there are no widely accepted benchmarks. So, we have built a private set of evaluations using Scale GenAI Platform to analyze the performance of Llama 3.1 on these enterprise use cases. These evaluations include enterprise-style chatbot prompts for financial services, legal, and edTech.
In addition to unbiased benchmarks, enterprises also need a platform to manage evaluations of LLMs. Scale GenAI Platform is the perfect enterprise testing ground for closed- and open-source models as it has built-in evaluation capabilities that enable users to easily compare results across different models, along with application development and deployment functionality.
<h3>The Setup</h3>
We evaluated the performance of Llama 3.1 405B across thousands of prompts and compared its performance to the best closed and open-source models across several enterprise domains:
<ol>
<li dir="ltr" aria-level="1">
Enterprise-style chatbot prompts covering legal, edTech, and financial services use cases. We grouped prompts into two different categories: short (1 - 2 sentences) and complex (3 - 5 sentences).
</li>
<li dir="ltr" aria-level="1">
Synthesis from content&nbsp;including legal, financial services, biology, and history domains
</li>
</ol>
<h3>Datasets</h3>
To start, we developed custom knowledge bases comprised of finance and legal textbooks, as well as other source materials that were relevant to each domain, such as legal codes. These datasets consisted of what would be considered general knowledge for each domain.
These knowledge bases were then imported into Scale GenAI Platform to generate evaluation datasets for each of the domains. For each domain, we generated chatbot-style datasets that simulated lawyer queries to a legal chatbot, financial services employee queries to a finance chatbot, and student queries to an edTech chatbot. Each model was also provided relevant context (chunks of text) from a data source on which it should ground its answer.&nbsp;
For real-world enterprise engagements, we use proprietary source documents and relevant industry data to generate custom evaluation datasets. For enterprise engagements, we would often include a gold standard &ldquo;expected output&rdquo; for each query which would then be used to evaluate model accuracy. For the purposes of our evaluations of Llama 3.1, we focused on head-to-head model comparisons only.
<h3>Performing the Evaluations</h3>
We then relied on human experts to evaluate models head-to-head in Scale GenAI Platform and created a weighted average score across multiple factors, such as which model provided the most complete, accurate, or relevant answer. The table below summarizes this rubric and the criteria used to determine a superior response.&nbsp;
<div dir="ltr" align="left">
<table><colgroup><col width="155"><col width="469"></colgroup>
<tbody>
<tr>
<td>
Completeness
</td>
<td>
Does the response fully answer all explicit aspects of the prompt? Does the response omit any essential information?
</td>
</tr>
<tr>
<td>
Accuracy
</td>
<td>
Which response contains the most accurate and reliable information, and which aligns with established facts or evidence? Established facts can be from common knowledge or verified using the provided context.
</td>
</tr>
<tr>
<td>
Depth
</td>
<td>
Which response provides a higher level of detail, insight, and nuance?
</td>
</tr>
<tr>
<td>
Relevance
</td>
<td>
Which response provides the most useful supporting information and claims in answering the main question or prompt? The supporting information logically defends or clearly illustrates the key points and the central claims made in the response.
</td>
</tr>
<tr>
<td>
Clarity
</td>
<td>
Which response is more clearly worded and understandable?
</td>
</tr>
<tr>
<td>
Formatting
</td>
<td>
Which response organizes the information in a way that supports the key points with clarity and brevity?
</td>
</tr>
</tbody>
</table>
</div>
We leveraged Scale GenAI Platform to perform these evaluations with human experts across thousands of examples to measure model performance using our enterpise rubrics. Expert evaluators were presented with the prompts and the response from two models and were asked to identify which model performed better in the different categories of the evaluation rubric.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=4eb9aea8b1fe1b38af4e9f9d4c67d7cd.png&amp;f=webp&amp;q=75" alt="" width="1600" height="799">Evaluation interface in Scale GenAI Platform
In addition to the prompt and individual model responses, human raters had access to the context used to generate the query. This context is useful for determining the accuracy of model responses and maps closely to how these models will perform at summarizing source material as part of a retrieval augmented generation (RAG) workflow.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=34a528e41bfb9877687518dab4705455.png&amp;f=webp&amp;q=75" alt="Example of the context provided for each evaluation within Scale GenAI Platform" width="1771" height="1411">
Example of the context provided for each evaluation within Scale GenAI Platform
<h3>The Results</h3>
Our evaluation datasets included thousands of queries for domains including financial services, legal, and edTech. After completing the evaluations, we aggregated the responses and generated a weighted average score across the rubric criteria and ranked models for performance in each query and, more broadly, for each evaluation dataset. We implemented controls for <a href="https://scale.com/blog/irr">inter-rater reliability</a> along with audits to ensure that we had reliable scores for each task and evaluation set.
From our evaluations, Llama 3.1 rivals the performance of many closed-source models and outperforms all other open-source models. Llama 3.1 performs particularly well in the categories of completeness, accuracy, and depth. Demonstrated common knowledge from the model is not as extensive as leading closed-source models but is quite close. That being said, Llama 3.1 is very good at summarizing content from the provided context, which is especially useful for enterprise workflows that rely on retrieval augmented generation (RAG).&nbsp;
Let&rsquo;s now examine a couple of detailed evaluation examples to demonstrate how Llama 3.1 405B stacks up.
<h4 dir="ltr">Llama 3.1 405B vs. Leading Closed-Source Models</h4>
In the example below, there is a legal domain question (&ldquo;User Prompt&rdquo;) followed by the model responses from Llama 3.1 405B on the left and a leading closed-source model on the right. Llama 3.1 performs favorably in accuracy and completeness compared to the closed-source model, as noted by the human preference rankings of &ldquo;neutral&rdquo; (meaning there is an equal preference for the responses from each model) for the categories of accuracy, completeness, and depth.
Qualitatively, Llama 3.1 405B is able to provide a clear, well-structured, and thorough response to the query. The model accesses the context (&ldquo;Reference Material&rdquo;) and synthesizes its answers in reference to the most relevant portions. For example, the model response cites the specific rule 4.10.3: &ldquo;This regime is based on Listing Rule 4.10.3, which makes it a requirement for listed entities to comply with the recommendations or provide an explanation for non-compliance.&rdquo; Other instances of these references are highlighted in the image. Because the closed source model also references this context well, the evaluation earned a neutral rating from human raters.&nbsp;
The ability to reference context and provide clear and succinct summaries is critical for not just legal use cases, but many enterprise use cases in any industry. Retrieval Augmented Generation (RAG) workflows rely on completion models like Llama 3.1 405B to be able to reliably summarize the relevant context with accuracy.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=e64027bca5d0ec5d34c52432b4c40b3d.png&amp;f=webp&amp;q=75" alt="" width="1600" height="1518">
&nbsp;
<h4 dir="ltr">Llama 3.1 405B vs. Leading Open-Source Models</h4>
In the finance example below, we evaluate responses from Llama 3.1 405B on the left and&nbsp; Llama 3 70B on the right. Llama 3.1 405B outperforms Llama 3 70B (as well as all other open-source models) in accuracy, completeness, and depth per our human evaluators.
Llama 3 70B is already quite good at many tasks, especially for a smaller model (see rankings on the <a href="https://scale.com/leaderboard">SEAL Leaderboard</a>, where this model outperforms many larger closed-source models). However, Llama 3.1 405B&rsquo;s responses stand out in a few ways. The response is more complete and robust, often including additional steps that Llama 3 70B does not. It also is more useful, for example by breaking down the cost-of-capital calculation and separating it into bullets, while Llama 3 70B only includes some of these details tucked away in a paragraph. Better formatting and more complete explanations make it much easier for users to quickly understand core concepts.&nbsp;
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=3c101283c062cb16d749ae6003e985cb.png&amp;f=webp&amp;q=75" alt="" width="1491" height="1600">
<h3>Conclusion</h3>
These are just a couple of examples out of the thousands of responses that we evaluated where Llama 3.1 stood out as the clear leader of open-source models for enterprise applications and&nbsp; having comparable performance to leading closed-source models.
While we focused on legal, financial services, and edTech in our evaluations, similar performance can be expected across a wide range of other enterprise domains including telecommunications, insurance, customer service, and more. As Llama 3.1 is open-source, it can be customized and leveraged for enterprise tasks for a lower cost of entry compared to most closed-source models.&nbsp;
With the evaluation capabilities in Scale GenAI Platform, your enterprise can get an unbiased sense of how leading LLMs such as Llama 3.1 perform for your specific use case. You can further customize the model via retrieval augmented generation (RAG) and fine-tuning, and deploy these models to production, including in your own virtual private cloud (VPC).To see how well Llama 3.1 performs for your unique enterprise use case using <a href="https://scale.com/genai-platform">Scale GenAI Platform</a>, book a demo below today.

Get started with Llama 3.1

Meta looked to Scale to evaluate the enterprise capabilities of Llama 3.1 405B with Scale GenAI Platform.

Meta and Scale Partner on Enterprise Adoption of Llama 3.1

Meta and Scale Partner to Drive Enterprise Adoption of Llama 3.1 405B Using Scale GenAI Platform

Today, we are excited to announce a multi-year strategic partnership with Amazon Web Services (AWS) to accelerate enterprise and public sector generative AI adoption. With an exponentially growing number of enterprise and public sector organizations looking to incorporate generative AI (GenAI) to augment their workforce, challenges remain in successfully deploying this technology in production.&nbsp;
In our experience, customer GenAI initiatives often stall after proof of concept because they do not trust and are unable to customize GenAI models with their proprietary data. That is why we are excited to partner with AWS to be their first model customization and evaluation partner to accelerate the GenAI journey. Through this collaboration, AWS customers will be able to leverage the&nbsp;<a href="https://aws.amazon.com/marketplace/pp/prodview-3odiiiv22ydjc?sr=0-3&amp;ref_=beagle&amp;applicationId=AWSMPContessa">Scale GenAI Platform</a> with the AWS Generative AI Innovation Center or on the AWS Marketplace to systematically evaluate their GenAI apps and leverage those insights to further customize, configure, and fine-tune the underlying models &ndash; all on top of Amazon Bedrock.&nbsp;
In addition to evaluating and fine-tuning models, Public Sector customers can access <a href="https://aws.amazon.com/marketplace/pp/prodview-e7onk7j7ju6cw?sr=0-1&amp;ref_=beagle&amp;applicationId=AWSMPContessa">Scale Donovan</a> through the AWS Marketplace. Donovan is an AI platform built for government organizations with federal security requirements in mind. Donovan uncovers insights and expedites traditional workflows to drive mission success. Public sector teams can leverage existing AWS infrastructure services to host Donovan and access leading GenAI models on secure networks. Our commercial customers have realized increased worker productivity by implementing AI in production systems, and with Donovan, the public sector can also achieve better mission outcomes.
To learn more about the partnership, visit our Scale&nbsp;<a href="https://aws.amazon.com/marketplace/seller-profile?id=seller-j2pcb7tbmx6du">AWS partner page</a> or engage with the <a href="https://aws.amazon.com/ai/generative-ai/innovation-center/">AWS Generative AI Innovation Center</a> which pairs you with AWS strategy and science experts.

Amazon Web Services and Scale partner to accelerate enterprise and public sector Generative AI adoption. 

AWS + Scale Partner to Accelerate Generative AI Adoption

AWS + Scale Partner to Bring Generative AI to Enterprises and Public Sector Customers

All of our external success&mdash;powering breakthroughs in L4 autonomy, partnering with OpenAI on RLHF going back to GPT-2, supporting the DoD and every major AI lab, and the recent $1bn financing transaction&mdash;all of it is downstream from us hiring the best people for the job. Talent is our #1 input metric.
Because of this, I spend a lot of my time on recruiting. I either personally interview every hire or sign off on every candidate packet. It&rsquo;s the thing I spend the plurality of my time on, easily. But everyone can and should contribute to this effort. There are almost a thousand of us now, and it takes a lot to hire quickly while maintaining, and continuing to raise, our bar for quality.
That&rsquo;s why this is the time to codify a hiring principle that I consider crucial to our success:&nbsp;
Scale is a meritocracy, and we must always remain one.
Hiring on merit will be a permanent policy at Scale.&nbsp;
It&rsquo;s a big deal whenever we invite someone to join our mission, and those decisions have never been swayed by orthodoxy or virtue signaling or whatever the current thing is. I think of our guiding principle as MEI: merit, excellence, and intelligence.
That means we hire only the best person for the job, we seek out and demand excellence, and we unapologetically prefer people who are very smart.
We treat everyone as an individual. We do not unfairly stereotype, tokenize, or otherwise treat anyone as a member of a demographic group rather than as an individual.&nbsp;
We believe that people should be judged by the content of their character &mdash; and, as colleagues, be additionally judged by their talent, skills, and work ethic.
There is a mistaken belief that meritocracy somehow conflicts with diversity. I strongly disagree. No group has a monopoly on excellence. A hiring process based on merit will naturally yield a variety of backgrounds, perspectives, and ideas. Achieving this requires casting a wide net for talent and then objectively selecting the best, without bias in any direction. We will not pick winners and losers based on someone being the &ldquo;right&rdquo; or &ldquo;wrong&rdquo; race, gender, and so on. It should be needless to say, and yet it needs saying: doing so would be racist and sexist, not to mention illegal.
Upholding meritocracy is good for business and is the right thing to do. This approach not only results in the strongest possible team, but also ensures we&rsquo;re treating our colleagues with fairness and respect.
As a result, everyone who joins Scale can be confident that they were chosen for their outstanding talent, not any other reasons.
MEI has gotten us to where we are today. And it&rsquo;s the same thing that&rsquo;ll get us where we&rsquo;re going, as we embark on our next chapter focusing on data abundance, frontier data, and reliable measurement to accelerate the development and adoption of AI models.
&nbsp;
&nbsp;

Scale is a meritocracy, and we must always remain one.

Meritocracy at Scale

With the rapid advancement of AI model capabilities, it is necessary, now more than ever, to test and evaluate AI systems to ensure that it is safe to deploy for its intended use case. While AI developers aim to make models safe, unforeseen vulnerabilities like model hallucinations or bias still exist. As the government looks to adopt and scale their use of AI, it is critical that they are acquiring and using safe, ethical and responsible AI systems.&nbsp;
Scale AI is committed to promoting AI safety. Through our test and evaluation offering, Scale Evaluation, customers can measure AI&nbsp; performance, safety, and reliability using industry-leading benchmarks. Leading commercial and public sector organizations including <a href="https://scale.com/blog/test-evaluation-vision">OpenAI</a> and <a href="https://scale.com/blog/scale-partners-with-cdao-to-test-and-evaluate-llms">CDAO</a> use Scale Evaluation to assess AI safety.&nbsp;
Scale&rsquo;s commitment to AI safety extends beyond our customers.&nbsp;
<ul>
<li dir="ltr" aria-level="1">
Scale played a critical role in shaping the best practices for software security by voluntarily signing the <a href="https://www.cisa.gov/securebydesign/pledge">Cybersecurity and Infrastructure Security Agency&rsquo;s Secure by design pledge.&nbsp;</a>
</li>
<li dir="ltr" aria-level="1">
Scale is collaborating with the <a href="https://scale.com/blog/scale-joins-us-ai-safety-consortium">National Institute of Standards and Technology</a> in the Artificial Intelligence Safety Institute Consortium.
</li>
<li dir="ltr" aria-level="1">
Scale contributed a red-teaming platform to the <a href="https://www.whitehouse.gov/briefing-room/statements-releases/2023/05/04/fact-sheet-biden-harris-administration-announces-new-actions-to-promote-responsible-ai-innovation-that-protects-americans-rights-and-safety/">White House-supported DEFCON 31</a> event.
</li>
<li dir="ltr" aria-level="1">
Scale launched the <a href="https://scale.com/blog/safety-evaluations-analysis-lab">Safety, Evaluations, and Alignment Lab (SEAL)</a>,&nbsp; a research initiative focused on increasing transparency and standardization in safety and compliance for deploying Large Language Models (LLMs) through the development of advanced evaluation and red teaming solutions.
</li>
<li dir="ltr" aria-level="1">
Scale published a <a href="https://scale.com/blog/measuring-mitigating-risk-wmdp">novel safety evaluation benchmark for LLMs</a>: the Weapons of Mass Destruction Proxy (WMDP) - measuring hazardous knowledge contained by LLMs within domains including biosecurity, chemical security, and cybersecurity.
</li>
</ul>
These partnerships have helped us improve and refine our test and evaluation approach and maintain the highest standard for AI evaluation. We built Scale Evaluation leveraging <a href="https://scale.com/guides/test-and-evaluation-vision">our deep expertise from developing, fine-tuning, and testing AI models for customers. </a>Scale Evaluation is underpinned by our <a href="https://scale.com/blog/test-and-evaluation-white-paper">T&amp;E white paper methodology</a> that defines an industry-leading technical framework for AI evaluation.
Scale Evaluation equips public sector organizations with the tools required to fulfill the requirements outlined by the Biden-Harris Administration&rsquo;s AI Executive Order and OMB Memorandum. By using the comprehensive suite of tools and resources within Scale Evaluation, agencies can identify AI risks, simulate and measure AI in real-world context, and understand model performance. Furthermore, Scale Evaluation facilitates the assessment and continuous improvement of AI systems, ensuring they are both effective and compliant with federal mandates.
As the landscape of AI evolves, Scale AI continues to define the frontier of safety and standardization in AI deployments. Our rigorous approach through Scale Evaluation, combined with strategic partnerships and initiatives like SEAL, underscores our unwavering commitment to advancing AI safety and reliability.<a href="https://scale.com/evaluation/public-sector"> Learn more about how Scale Evaluation can support your agency's responsible AI needs today.</a>

Responsible AI with Scale Evaluation for Public Sector

Responsible AI with Scale Evaluation for the Public Sector

Today marks the one-year anniversary of Donovan, our pioneering mission-focused AI application designed to responsibly support the public sector. Utilizing a variety of large language models (LLMs), Donovan has helped to inform the ways government organizations can leverage AI to make faster, more informed decisions and minimize time searching for information.&nbsp;&nbsp;
Over the past year, we have successfully rolled out Donovan to numerous government agencies. As we celebrate this milestone, we wanted to look back at the AI capabilities that have been deployed and what Scale AI Public Sector has planned for the next year to accelerate the adoption of AI to provide decision overmatch.&nbsp;
<ol>
<li dir="ltr" aria-level="1">
Information Retrieval: Donovan incorporates Retrieval-Augmented Generation (RAG) so that users can use LLMs to interact with information pertaining to their mission. This feature allows users to upload, access, and query data in secure classified environments. Donovan provides precise, well-informed insights that are directly linked to verifiable sources. Each insight generated by Donovan is accompanied by citations, grounding the information in credible references, and enhancing the reliability of the data presented.
</li>
<li dir="ltr" aria-level="1">
Geospatial Chat: Another standout feature of Donovan is geospatial chat, which integrates geographic filtering with the power of an LLM. This allows users to interact with a map interface where they can select a specific area and then pose questions directly related to that location. Donovan processes these queries and delivers answers that are exclusively relevant to the chosen geographic area. Additionally, if the citation of the response includes a specific location, Donovan can pin that location on the map. This pin comes enriched with important metadata from the citation, enhancing the depth of information available.&nbsp;
</li>
<li dir="ltr" aria-level="1">
Text-to-API: Donovan's text-to-API capabilities enable seamless integration with other applications or systems. Users can input queries in natural language, which Donovan then translates into API requests. This capability allows Donovan to fetch and relay information from connected systems back to the user through a RAG response - presenting the information in natural language. This streamlined process not only simplifies interactions with complex databases, systems, and other data sources but also ensures that users receive accurate and relevant information swiftly, enhancing productivity and decision-making. This includes making enormous and mission critical information available through our partnerships with Flashpoint, Strider, and 4DV.
</li>
<li dir="ltr" aria-level="1">
Report Generation: This feature allows users to create customized templates that cater to common requests for information or specific documents relevant to operations. Once a template is set up, users can generate reports that not only address predefined questions using LLMs but also consider new data as it becomes available. Donovan significantly reduces the manual effort required, allowing users to focus on strategic tasks while staying informed with the most up-to-date insights. As a result, users are able to cut their workflow from hours to minutes and spend more time on cognitive tasks that require human context and creativity.&nbsp;
</li>
</ol>
We are excited to continue improving Donovan to empower public sector organizations with AI. We&rsquo;ve listened to users and found that Donovan is most useful when tightly aligned with the tools, knowledge, and understanding of a team&rsquo;s mission. This insight shaped our vision for Donovan and guided our team to develop capabilities to expand Donovan&rsquo;s reach and empower even more organizations to leverage AI for mission-critical tasks.&nbsp;
Our vision for Donovan includes:
<ol>
<li dir="ltr" aria-level="1">
Role-Based Workflows: Donovan is positioned to utilize tools like RAG and seamlessly access public APIs. More than just a tool, Donovan will have the ability to suggest contextually appropriate questions and mimic specialized roles, assuming personas such as a geospatial imagery analyst or a cybersecurity specialist. These enhancements are designed to make Donovan not only a user-friendly platform but a personalized force multiplier, customized to the unique demands and challenges that public sector team members face.
</li>
<li dir="ltr" aria-level="1">
Frontier Multimodal AI Capabilities: In support of role-based workflows, Donovan will be able to utilize custom models that retain specialized knowledge and skills for specific tasks, and can outperform standard off-the-shelf models. For example, a fine-tuned model for analyzing military reports and communications could quickly extract and summarize key information, understand DOD-specific acronyms, and provide actionable insights that strictly follow military doctrine. Donovan will continue to elevate the quality of insights for users through improved AI techniques. Semantic search enhances Donovan&rsquo;s ability to understand a user&rsquo;s intent from a query. Teams will also be able to explore different modalities with large multimodal models (LMMs). Users can leverage LMMs for a variety of use cases including identifying objects in geospatial imagery and reading and interpreting maps. 
</li>
</ol>
We are incredibly excited about the upcoming enhancements to Donovan. As exciting new features are being brought to life, Donovan can continue to help transform the way government organizations operate. If Donovan can support your team, contact us at <a href="mailto:federal@scale.com">federal@scale.com</a>

Highlighting the progress and vision of  Scale Donovan after launching one year ago.

Celebrating One Year of Scale Donovan

We now live in a world where there are hundreds of large language models. But as these models become increasingly powerful and widely deployed, comparison of their performance and safety remains opaque.&nbsp;
As a third-party model evaluator trusted by leading AI labs, Scale is excited to release the <a href="https://scale.com/leaderboard">SEAL Leaderboards</a>, which rank frontier LLMs using curated private datasets that can&rsquo;t be gamed. Developed by our Safety, Evaluations, and Alignment Lab (SEAL) and assessed by verified domain experts, these leaderboards are regularly updated to include new models and capabilities. The SEAL Leaderboards maintain neutrality and integrity, ensuring rankings are tamper-proof and provide a true measure of model performance.
The initial domains covered include:
<ul>
<li dir="ltr" role="presentation">Coding</li>
<li>Instruction Following</li>
<li dir="ltr" role="presentation">Math (based on <a href="https://arxiv.org/abs/2405.00332">GSM1k</a>)</li>
<li dir="ltr" role="presentation">Multilinguality</li>
</ul>
We hope this initiative advances transparency and openness in the development and evaluation of frontier models, which will in turn accelerate the adoption of AI.&nbsp;
<blockquote>
Companies are incentivized to game leaderboards in order to one-up one another. This makes it hard to tell whether AI systems are actually improving. That&rsquo;s why it&rsquo;s important that organizations such as Scale assess these AI systems with their private evaluations.
</blockquote>
&mdash; Dan Hendrycks, Director of the Center for AI Safety
<blockquote>
The work Scale is doing to evaluate the performance, reliability, and safety of AI models is crucial. Government agencies and the general public alike need an independent, third-party like Scale to have confidence that AI systems are trustworthy and to accelerate responsible AI development.
</blockquote>
&ndash; Dr. Craig Martell, Former Chief Digital and Artificial Intelligence Officer (CDAO) for the U.S. Department of Defense
While progress has already been made to evaluate the performance of frontier models by the research community, there are some issues that Scale is uniquely positioned to solve. These challenges include:
<ul>
<li dir="ltr" aria-level="1">
Contamination &amp; Overfitting: There is a lack of high quality, trustworthy evaluation datasets that have not been contaminated and shared widely.
</li>
<li dir="ltr" aria-level="1">
Inconsistent Reporting: There is a lack of consistency in model comparisons and reliability in how eval results are reported.
</li>
<li dir="ltr" aria-level="1">
Unverified Expertise: There is a lack of rigorous assessment of evaluators&rsquo; expertise for specific domains (e.g. coding) in community-based evaluations.
</li>
<li dir="ltr" aria-level="1">
Inadequate Tooling: There is a lack of good product tooling for understanding and iterating on eval results to improve models without overfitting.
</li>
</ul>
It is with these problems in mind that Scale launched our <a href="https://scale.com/blog/safety-evaluations-alignment-lab">SEAL</a> research lab last November, aimed at enhancing evaluation quality, transparency, and standardization for frontier models.
The SEAL Leaderboards are a set of LLM model rankings across a number of popular public models, based upon curated private datasets that can&rsquo;t be gamed, all funded and developed by Scale. Compared to existing benchmarks and community driven approaches, we place a high emphasis on:
<ul>
<li dir="ltr" aria-level="1">
Leaderboard Integrity1: Unlike most public benchmarks, Scale's proprietary datasets will remain private and unpublished, ensuring they cannot be exploited or incorporated into model training data. We limit entries to the SEAL Leaderboards from AI developers who may have seen the specific prompt sets via API logging, ensuring unbiased evaluations. To ensure rigorous results and maintain accountability, we plan to collaborate with trusted third party organizations to help review our work.
</li>
<li dir="ltr" aria-level="1">
Domain Specialization: The SEAL Leaderboards cover a range domains including coding, instruction following, math, and multilinguality. For each domain, experts create prompt sets from scratch, and tailor the evaluation methods to what we believe works best for that specific area. We ensure that evaluators are qualified domain experts, including via domain-specific interviews and tests, such as accurately completing golden tasks with known answers.
</li>
<li dir="ltr" aria-level="1">
Data Quality: Both the prompts and the ratings have undergone extensive, multi-round reviews and passed internal QAs, to ensure that the results are trustworthy and high quality. To ensure rigorous results and maintain accountability, we plan to collaborate with trusted third party organizations to help review our work.
</li>
<li dir="ltr" aria-level="1">
Methods Transparency: We publish details about our <a href="https://scale.com/leaderboard/methodology">evaluation methodology</a>, along with key insights from our evaluation runs, beyond just the raw SEAL Leaderboard rankings.&nbsp; We welcome community feedback to refine our methods further.
</li>
</ul>
<table style="border-collapse: collapse; width: 99.9754%; height: 92.2745px;" border="1"><colgroup><col style="width: 49.9115%;"><col style="width: 49.9115%;"></colgroup>
<tbody>
<tr style="height: 18.4549px;">
<td style="height: 18.4549px;">
<h3>Evaluation Challenge</h3>
</td>
<td style="height: 18.4549px;">
<h3>SEAL Solution</h3>
</td>
</tr>
<tr style="height: 18.4549px;">
<td style="height: 18.4549px;">Contamination &amp; Overfitting</td>
<td style="height: 18.4549px;">Private Unexploitable Datasets</td>
</tr>
<tr style="height: 18.4549px;">
<td style="height: 18.4549px;">Inconsistent Reporting</td>
<td style="height: 18.4549px;">Transparent &amp; Consistent Methodologies</td>
</tr>
<tr style="height: 18.4549px;">
<td style="height: 18.4549px;">Unverified Expertise</td>
<td style="height: 18.4549px;">Vetted Domain Experts</td>
</tr>
<tr style="height: 18.4549px;">
<td style="height: 18.4549px;">Inadequate Tooling</td>
<td style="height: 18.4549px;">
<div class="docos-replyview-body docos-anchoredreplyview-body docos-replyview-body-emoji-reactable docos-replyview-body-emoji-reactable-background" dir="ltr">Scale Evaluation Platform</div>
</td>
</tr>
</tbody>
</table>
In an effort to drive improved evaluation across the AI community, we plan to continue launching new <a href="https://scale.com/leaderboard">SEAL Leaderboards</a> covering additional domains and capabilities, refreshing the prompt sets, and adding in new frontier models as they become available. As the industry develops and new models are released, we aim to refresh these leaderboards multiple times a year, ensuring that the rankings remain up-to-date and reflective of the latest advancements. This approach underpins the first truly expert-driven, trustworthy LLM contest open to all. Finally, in addition to the SEAL Leaderboards, we are also announcing the general availability of Scale Evaluation: a platform to enable AI researchers, developers, enterprises, and public sector organizations to analyze, understand, and iterate on their AI models and production applications. Learn more&nbsp;about Scale Evaluation <a href="http://scale.com/evaluation/model-developers">here</a>.
Today's announcement marks a significant milestone in Scale's mission of accelerating the development of AI applications. As we continue to build upon the SEAL Leaderboards, we remain dedicated to working closely with the research community, AI developers, and other stakeholders to address the most pressing challenges of AI evaluation in service of enabling the potential of AI models.
If you have feedback on the SEAL Leaderboards, would like to run your model against these evaluations, or would like to be included in future leaderboards, we&rsquo;d love to hear from you at&nbsp;<a href="mailto:seal@scale.com">seal@scale.com</a>. 
&nbsp;
<hr>
1To maintain the integrity of the SEAL Leaderboards, by default, models are only eligible for inclusion the first time their developers have one of their models prompted on the evaluation prompts. In exceptional cases, such as major launches where we believe the overfitting risks are low, we may include newly launched models even if the AI developers may have previously seen the prompt set in API logs. In such instances, the new model will be included but grayed out on the leaderboard, with its overfitting risks clearly noted. Additional detail on how these datasets were created and evaluated is available alongside each leaderboard.

Scale is excited to release the SEAL Leaderboards, which rank frontier LLMs using curated private datasets.

Scale's SEAL Leaderboards

Scale's SEAL Research Lab Launches Expert-Evaluated LLM Leaderboards

For 8 years, Scale has been the leading AI data foundry helping fuel the most exciting advancements in AI, including autonomous vehicles, defense applications, and generative AI.
Today, we&rsquo;re announcing that Scale has closed a $1B financing transaction at a $13.8B valuation. The financing, a mix of primary and secondary, is led by existing investor Accel with nearly all of our existing investors participating: Y Combinator, Nat Friedman, Index Ventures, Founders Fund, Coatue, Thrive Capital, Spark Capital, NVIDIA, Tiger Global Management, Greenoaks, and Wellington Management. We also are excited to welcome new investors: Cisco Investments, DFJ Growth, Intel Capital, ServiceNow Ventures, AMD Ventures, WCM, Amazon, Elad Gil, Meta, and Qualcomm Ventures.&nbsp;
With this milestone, here&rsquo;s where we are on our journey and what comes next.
<h2 dir="ltr">Becoming the Data Foundry for AI</h2>
In 2016, I was studying AI at MIT. Even then, it was clear that AI is built from three fundamental pillars: data, compute, and algorithms. I founded Scale to supply the data pillar that advances AI by fueling its entire development lifecycle.&nbsp;
Over the past 8 years, Scale has powered nearly every major breakthrough field of AI:
<ul>
<li dir="ltr" aria-level="1">
Scale&rsquo;s Autonomy Data Engine powered breakthroughs in L4 autonomy.
</li>
<li dir="ltr" aria-level="1">
Scale&rsquo;s Public Sector Data Engine has powered many major AI programs within the US Department of Defense.
</li>
<li dir="ltr" aria-level="1">
Scale partnered with OpenAI in the first experiments of <a href="https://openai.com/index/fine-tuning-gpt-2/">reinforcement learning with human feedback (RLHF) on GPT-2</a>, and scaled these techniques to <a href="https://openai.com/index/instruction-following/">InstructGPT</a> and beyond.
</li>
<li dir="ltr" aria-level="1">
Scale worked on the <a href="https://www.whitehouse.gov/briefing-room/statements-releases/2023/05/04/fact-sheet-biden-harris-administration-announces-new-actions-to-promote-responsible-ai-innovation-that-protects-americans-rights-and-safety/">White House</a>-supported DEFCON 31 red-teaming event and with the U.S. <a href="https://scale.com/blog/scale-partners-with-cdao-to-test-and-evaluate-llms">Department of Defense</a> on rigorous evaluation, testing, and red-teaming of LLMs.
</li>
</ul>
Today, Scale supplies data to power nearly every leading AI model, serving organizations like OpenAI, Meta, Microsoft, and more.
<h2 dir="ltr">The Next Phase: Data Abundance for Frontier AI</h2>
Major problems in AI data still remain. The scaling laws imply an exponentially growing need for data as models get bigger, which raises a key question: will we run out of data?
Just as data, compute, and algorithms comprise the three pillars of AI, we believe the future of AI data in turn rests on three principles:
<ul>
<li dir="ltr" aria-level="1">
Data Abundance: We must build the data foundry that ushers in an era of AI-ready data abundance, and not resign ourselves to data scarcity.&nbsp;
</li>
<li dir="ltr" aria-level="1">
Frontier Data: As we develop progressively more powerful AI, we must build frontier data which is always pushing the boundaries of AI capabilities towards complex reasoning, agents, multimodality, and more.
</li>
<li dir="ltr" aria-level="1">
Measurement and Evaluation: We must build an evaluation system that enables measurement of AI to build confidence, drive adoption, and scale impact.
</li>
</ul>
Abundance is not the default; it&rsquo;s a choice. It requires bringing together the best minds in engineering, operations, and AI.
Our calling is to build the data foundry for AI, and with today&rsquo;s funding, we&rsquo;re moving into the next phase of that journey &ndash; accelerating the abundance of frontier data that will pave our road to AGI.
There&rsquo;s a lot left to do. If this challenge excites you, <a href="https://scale.com/careers">join us</a>.

Scale supplies data to power nearly every leading AI model. 

Scale’s Series F: Expanding the Data Foundry for AI

When Scale introduced Donovan, a large language model (LLM) platform built for the public sector, one of the top questions we were asked was, &ldquo;Why Donovan?&rdquo; For Scale, Donovan is not just a name, it is a tribute to the visionary Major General William &ldquo;Wild Bill&rdquo; Donovan, the founder of the Office of Strategic Services (OSS) the precursor to the modern Central Intelligence Agency &ndash; and the father of modern intelligence in the United States. Donovan&rsquo;s life and ethos embodies the qualities we value most, particularly innovation and a forward-looking vision.&nbsp;
Major General Donovan&rsquo;s background spanning the Department of Defense (DoD), the Intelligence Community (IC), and civilian roles mirrors the range of organizations that Scale's Donovan has been designed to support. His legacy of innovation and leadership has long been an inspiration to many Public Sector Scaliens, many of whom served in the DoD and IC themselves. Naming Scale&rsquo;s flagship LLM platform for the Public Sector was a fitting tribute to &ldquo;Wild Bill.&rdquo;
Just as Major General Donovan revolutionized intelligence gathering and operations, Scale&rsquo;s Donovan aims to transform the public sector with large language models to enable mission success. With a model and infrastructure agnostic architecture, Scale Donovan provides users unmatched flexibility to seamlessly select from a variety of state-of-the-art models, including those provided by the customer, that can be fine tuned to meet specific operational requirements.
Donovan can be made available to organizations at any classification level and is currently deployed across various classified and unclassified networks. This includes an Impact Level 4 (IL4) provisional authorization from the Defense Information Systems Agency (DISA) and a FedRAMP High &ldquo;in process&rdquo; designation from the Joint Authorization Board (JAB). Scale Donovan&rsquo;s APIs and advanced retrieval augmented generation (RAG) capabilities empower users to uncover mission-critical insights and information from the thousands of available documents and datasets. Scale Donovan&rsquo;s RAG capabilities enable users to converse effectively with their documents by asking questions in natural language and receiving precise answers with citations.&nbsp;
Hope you enjoyed learning more about Donovan the person. Now learn more about <a href="https://scale.com/donovan">Scale&rsquo;s Donovan here.&nbsp;</a>

Meet the Legend Behind the Name - Scale Donovan

It is no secret that the world is heavily investing in Artificial Intelligence (AI) because of the economic and societal benefits that it has the potential to deliver. Last year, President Biden proposed roughly $5.5 billion in government spending and his most recent budget proposal once again prioritized AI investment to ensure the United States&rsquo; global leadership in the government&rsquo;s adoption of AI. Congress has also done its part by funding critical elements, such as the National Institute of Standards and Technology&rsquo;s (NIST) recently announced <a href="https://www.nist.gov/artificial-intelligence/artificial-intelligence-safety-institute">AI Safety Institute Consortium</a> that will bring industry together to develop essential AI safety benchmarks, frameworks and techniques.&nbsp;
Despite the attention that AI is receiving, much of the funding still prioritizes early-stage research, incremental improvements for existing programs, or short-term pilot projects. While these efforts are important, Congress must also fund new, potentially transformative AI initiatives that will unlock the potential economic and efficiency gains.
We cannot afford to wait to invest sufficiently in AI. As Congress begins to debate the FY25 Authorizations and Appropriations, Scale recommends addressing three key items to ensure that the United States remains the global leader in the government&rsquo;s adoption of AI and to yield the maximum value of its investments for American taxpayers.&nbsp;
<h3 dir="ltr" role="presentation">&nbsp;</h3>
<h3 dir="ltr" role="presentation">Recommendation 1: The government must start to adequately plan for AI costs to truly implement AI.&nbsp;</h3>
If the United States wants to lead the world in the development and deployment of AI across the federal government, then it must start by properly planning for AI program costs, such as training data, in the same way that it plans for more traditional costs like hardware and maintenance.
AI has the potential to improve the efficiency of nearly every government use case. In a matter of years, it is likely to be used in every program in the U.S. government, from building more high-quality algorithms for Department of Defense autonomy programs to accelerating regulatory actions, such as speeding up the rulemaking process. To achieve this vision, however, the President and Congress must include the ongoing sustainment costs for AI in the budgets. AI systems require constant high-quality data to train, maintain, and improve foundational models so the technology performs reliably for its intended use case. For this reason, the commercial industry has long invested in &ldquo;AI-ready data&rdquo; to consistently train and maintain its AI models. While the federal government has long considered AI-ready data to be a &ldquo;national asset,&rdquo; the acquisition system, which was set up around maintaining platforms like ships and airplanes, is not well suited to fund a different type of technology that requires ongoing refinement for high-quality outputs.
To truly harness the power of AI, Congress and the White House must create and fund dedicated AI lines in each program that will create, maintain, and enhance effective AI capabilities. &nbsp;
<h3 dir="ltr" role="presentation">Recommendation 2: The government must prioritize implementing and fielding AI, in addition to research.</h3>
There is no doubt that some fundamental research gaps exist, particularly around AI safety, and we must adequately fund agencies, such as NIST, to do this crucial work. However, If the U.S. government is serious about deriving economic value and efficiency gains from AI, then it must invest in putting AI capabilities into users&rsquo; hands so its true benefits can be realized.
Historically, keeping up with the pace of emerging technologies, like AI, has been a challenge for the U.S government, primarily because funding cycles require budgets to be crafted, in some cases, 18 months before they are approved. Beyond the budget cycle, acquisition programs were designed decades ago to procure legacy systems. For example, the funding levels contained in the FY24 budget, which Congress recently passed in March, 2024, were largely set before ChatGPT thrust the idea of generative AI into the national consciousness. In reviewing FY24 spending, and as more details related to the FY25 budget become available, it is clear that there is a strong, continued, emphasis on funding AI research and development and short-term pilot programs. While this can be valuable, as history has shown, this strategy often fails at getting AI into the hands of government users and warfighters in a meaningful way.&nbsp; In the past five years, only one AI effort has fully transitioned from a pilot program to a productive program of record.
&nbsp;
<h3 dir="ltr" role="presentation">Recommendation 3: More targeted investment is needed for AI programs, instead of using AI funds on legacy software programs.&nbsp;</h3>
Despite the at least $5 billion per year in reported topline AI funding, the amount of funding available for AI programs is less than it seems as the majority of that money typically is used to fund legacy software and hardware, which may only include a small use of AI. For example, if a program is receiving $350 million for its entire annual budget, but only $10 million of that funding is geared towards AI development and deployment, all $350 million is reported as AI spend.&nbsp;
There is no shortage of newly launched AI initiatives that have been established across the government. However, the programs are underfunded to achieve the intended outcome. To adequately fund these programs, it is vital that the reported AI funding is going to carry out actual, dedicated AI programs. To do so, Scale recommends that the U.S. government ensures that the entirety of its AI investment truly works to fund AI programs and efforts.
&nbsp;

Essential Elements to Ensure Effective AI Adoption in the U.S. Government 

In July 2023 we <a href="https://scale.com/blog/custom-llms">announced</a> the launch of <a href="https://llm-engine.scale.com">LLM Engine</a>, our open-source platform for customizing and serving large language models (LLMs). We decided to open-source this platform to give all enterprises the opportunity to accelerate their ability to host and fine-tune LLMs.
For customers with larger-scale hosting needs and more powerful fine-tuning features, we also built an advanced version of LLM Engine that is available on <a href="https://scale.com/genai-platform">Scale GenAI Platform</a>. Through deploying customized LLMs for customers we have found that the details of hosting LLMs are still obscure, so we wanted to shine a light on how this works and how LLM Engine is architected to optimize each step of the process.
And we are excited to announce that coming soon, <a href="https://scale.com/genai-platform">Scale GenAI Platform</a> customers can leverage Nvidia NIM microservices through LLM Engine for optimized LLM inference. Nvidia NIM microservices provide flexibility and performance for running customized generative AI in production.
In general, deploying LLMs to the cloud is an expensive operation that requires experience in both cloud infrastructure and machine learning. While great technologies exist for general service orchestration (e.g kubernetes), there are some distinct frictions with serving deep-learning workloads:
<ul>
<li dir="ltr" aria-level="1">
Effective GPU utilization - It is well known in the industry that there is a fierce GPU supply crunch. Squeezing as many queries per second (QPS) as possible out of your GPU is essential for managing hosting costs for models.
</li>
<li dir="ltr" aria-level="1">
MLE-friendly APIs - Given the incredible rate of ML advancement, ML engineers/researchers should focus on ML models. We do not want to waste precious engineering cycles on scaling models to a high QP and large-scale cloud environment. These activities should be built into a platform and easily automated with minimal effort from ML engineers.
</li>
<li dir="ltr" aria-level="1">
Experimentation speed - In most eng orgs, teams typically only maintain a few services that serve business functions from their team. In ML orgs, a single ML engineer often rapidly deploys several models as they experiment or as they deliver models from project to project. An ML Engineer's (MLE's) overall productivity should not slow down, even when the number of endpoints they manage grows.
</li>
<li dir="ltr" aria-level="1">
Efficient auto-scaling - While QPS is a good metric for individual GPU nodes, ML teams and production applications also need the number of GPU nodes serving a model to autoscale up and down efficiently as demand fluctuates. Because of the lack of availability and cost of GPU nodes, it is also necessary to autoscale down to zero to avoid a very high minimum cost when experimenting with a large number of models.
</li>
<li dir="ltr" aria-level="1">Data privacy concerns - While many commercial models are only available via API endpoints, when possible, we wanted to host OSS models on our own infrastructure to eliminate any privacy concerns related to sending our internal proprietary data to an external service to leverage these models.</li>
</ul>
Our team found no open-source software that satisfied all these needs. Given that ML is a core competency within Scale, we did not want to lock ourselves into proprietary solutions that could not be customized.
We built LLM Engine as an open-source platform to serve and fine-tune foundation models, whether you use Scale's hosted infrastructure or your own cloud infrastructure. The advanced version of LLM Engine powers our&nbsp;<a href="https://scale.com/genai-platform">Scale GenAI Platform</a>. Today, all of our internal ML models are deployed with LLM Engine.
While the current <a href="https://github.com/scaleapi/llm-engine">public-facing documentation</a> for LLM Engine primarily focuses on the LLM-specific APIs, this blog will explore the <a href="https://scaleapi.github.io/launch-python-client/">lower-level APIs</a> that our internal MLE&rsquo;s leverage.
<h3>Usage Summary</h3>
We start with a quick overview of core API concepts and showcase how MLEs leverage LLM Engine to host their <a href="https://scale.com/blog/fine-tune-llama-2">custom models</a>. We encourage readers to check the <a href="https://scaleapi.github.io/launch-python-client/" target="_blank" rel="noopener">API docs</a> for full details. Note that code examples will have references to Launch, which is how we refer to LLM Engine internally.
First, users create a <a href="https://scaleapi.github.io/launch-python-client/concepts/model_bundles/" target="_blank" rel="noopener">ModelBundle</a>. Users can think of ModelBundles as deployable assets (code + model weights) that are capable of making &ldquo;prediction&rdquo; requests. Users must also specify an expected request/response schema for the bundle. Should users need complete customization, there is also an option for users to bring a full image as a <a href="https://scaleapi.github.io/launch-python-client/guides/custom_docker_images/" target="_blank" rel="noopener">RunnableBundle</a>. Below is a simple example:
<pre class="language-python"><code>from launch import LaunchClient
from pydantic import BaseModel

class MyRequestSchema(BaseModel):
 x: int
 y: str

class MyResponseSchema(BaseModel):
 __root__: int

def my_load_predict_fn(model):
 def returns_model_of_x_plus_len_of_y(x: int, y: str) -&gt; int:
 """MyRequestSchema -&gt; MyResponseSchema"""
 assert isinstance(x, int) and isinstance(y, str)
 return model(x) + len(y)
 return returns_model_of_x_plus_len_of_y

def my_load_model_fn():
 def my_model(x):
 return x * 2

BUNDLE_PARAMS = {
 "model_bundle_name": "test-bundle",
 "load_predict_fn": my_load_predict_fn,
 "load_model_fn": my_load_model_fn,
 "request_schema": MyRequestSchema,
 "response_schema": MyResponseSchema
 "requirements": ["pytest==7.2.1", "numpy"], # list your requirements here
 "pytorch_image_tag": "1.7.1-cuda11.0-cudnn8-runtime",
 # additional fields omitted
}

client = LaunchClient(api_key=os.getenv("LAUNCH_API_KEY"))
client.create_model_bundle_from_callable_v2(**BUNDLE_PARAMS)</code></pre>
After a ModelBundle is registered, users can now create <a href="https://scaleapi.github.io/launch-python-client/concepts/model_endpoints/" target="_blank" rel="noopener">ModelEndpoints</a>. ModelEndpoints are actual deployments of a specified model bundle that can handle production traffic. Users specify various deployment parameters, such as min/max number of workers, as well as the compute resources such as the number of GPUs, CPUs, etc. A single model bundle can be used in several ModelEndpoints, whether they are synchronous, asynchronous, or streaming:
<pre class="language-python"><code># Creating an asynchronous endpoint
ENDPOINT_PARAMS = {
 "endpoint_name": "demo-endpoint",
 "model_bundle": "test-bundle",
 "cpus": 1,
 "min_workers": 0,
 "endpoint_type": "async",
}
endpoint = client.create_model_endpoint(**ENDPOINT_PARAMS)</code></pre>
That&rsquo;s it! Once an endpoint has status READY, an MLE&rsquo;s model is ready to serve production traffic. We show an example of submitting an async request:
<pre class="language-python"><code>def predict_on_endpoint(request: MyRequestSchema) -&gt; MyResponseSchema:
 endpoint_request = EndpointRequest(args=request.dict(), return_pickled=False)
 future = endpoint.predict(request=endpoint_request)
 raw_response = future.get()
 response = MyResponseSchema.parse_raw(raw_response.result)
 return response
request = MyRequestSchema(x=5, y="hello")
response = predict_on_endpoint(request)</code></pre>
<h3>Open-Source LLM hosting</h3>
Leveraging the core API routes mentioned previously, LLM Engine includes specialized /llm/* APIs optimized for fine-tuning and hosting open-source LLM models. With LLM Engine, we host a model zoo of popular open-source LLMs (e.g Llama 2 or Mixtral 8x7B) that product teams can immediately leverage to build Generative AI applications. Under the hood, LLM Engine leverages open-source frameworks like <a href="https://github.com/vllm-project/vllm" target="_blank" rel="noopener">vLLM</a> to maximize tokens/sec.
<pre class="language-python"><code>from llmengine import Completion

response = Completion.create(
 model="mixtral-8x7b",
 prompt="Hello! ",
 max_new_tokens=10,
)
print(response.output.text)</code></pre>
Check out the <a href="https://llm-engine.scale.com/guides/completions/" target="_blank" rel="noopener">public docs</a> for usage details for LLM-focused APIs.
<h3>LLM Engine Architecture</h3>
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=60843cd603312f22b12e1ddc303e887e.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="371" height="297">
<h3>LLM Engine Gateway</h3>
The Gateway is the central API server that orchestrates all incoming user requests. All CRUD operations involving API objects such as endpoints, bundles, etc., go through the Gateway first. API object metadata is stored in a Postgres-based datastore. Endpoint prediction requests (e.g., &ldquo;Tasks&rdquo;) are routed first through the Gateway. The Gateway simply routes requests to the correct endpoint for sync or streaming tasks. For async tasks, the Gateway pushes tasks to the correct queue.
<h3>Endpoint Builder</h3>
Endpoint creation can be a long-running process depending on the Bundle provided by users. Therefore, LLM Engine separated out this process from the core Gateway for improved request stability. The Endpoint Builder is implemented as a <a href="https://docs.celeryq.dev/en/stable/userguide/workers.html" target="_blank" rel="noopener">Celery</a> worker that receives endpoint CRUD requests from a predefined queue. Endpoint creation then goes through the following steps:&nbsp;
<ol>
<li dir="ltr" aria-level="1">
Create an endpoint image - For Cloudpickle Bundles, the Endpoint Builder must first launch a <a href="https://github.com/GoogleContainerTools/kaniko" target="_blank" rel="noopener">Kaniko</a> job to bake the Bundle into a deployable container. Future Endpoint creation requests with the same bundle do not require rebuilding. For RunnableImage Bundles, this build step is skipped.
</li>
<li dir="ltr" aria-level="1">
Create Kubernetes Resources - The Endpoint Builder will translate user-specified requests (e.g resource_request, num_workers) to the correct k8s resources required to deploy the Endpoint. For the most part, one can think of an LLM Engine Endpoint corresponding 1:1 to a k8s Deployment. Several additional k8s resources (e.g PodDisruptionBudget, HorizontalPodAutoscaler) are also created, but we won&rsquo;t go into all of them in this post.
</li>
<li dir="ltr" aria-level="1">
(Async Endpoints Only) Create Queue - For async endpoints, an SQS queue is created to store EndpointRequests before they are consumed.&nbsp;
</li>
</ol>
<h3>Anatomy of an Endpoint</h3>
Pictured below is a bit more detail for an individual Endpoint worker:
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=2e7294bdc04835d13148bfec18f38b30.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="485" height="172">
Every worker for an endpoint is deployed as a k8s pod. Within the pod, there are two containers:
<ol>
<li dir="ltr" aria-level="1">
The User Container is the image built in the previously mentioned Endpoint creation process. For Cloudpickle Bundles, LLM Engine bakes the code into a Rest Framework. For RunnableImage Bundles where users essentially provide their own image, the API contract simply requires an HTTP listener at port 5005.
</li>
<li dir="ltr" aria-level="1">
The Forwarder Container contains LLM Engine system code required to forward requests to the User Container. For async endpoints, the forwarder handles the Celery logic to poll/forward tasks to the User Container and then persist results to a Celery Backend, which is S3 in our internal deployment. The Forwarder also emits metrics useful for Endpoint monitoring and can perform pre/post-processing hooks for incoming Predict requests.
</li>
</ol>
By leveraging a separate container LLM Engine isolates system dependencies and logic from user code, enabling the same bundle to be leveraged for sync/async/streaming endpoints without any dependency conflicts or any changes in user logic. Separation also enables users to have the option of complete customization should they need to bring their own image.
<h3>Autoscaling</h3>
MLE&rsquo;s can specify how to autoscale their endpoints with the requests_per_worker field. For example, a value of 2 would mean that each worker should handle about ~2 concurrent requests. Users can also specify the min_worker and max_worker they are willing to scale in order to balance costs with reliability.
<h3 dir="ltr">Sync/Streaming</h3>
To autoscale sync/streaming endpoints, LLM Engine leverages the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-adapter" target="_blank" rel="noopener">Prometheus Adapter</a> to create custom metrics for Horizontal Pod Autoscalers. Specifically, LLM Engine creates a request-concurrency-average metric whose target value is equal to the user-provided requests_per_worker value. To compute this custom value, LLM Engine leverages Istio&rsquo;s Request Duration Metric (e.g istio_request_duration_milliseconds) to approximate the average concurrency per endpoint.
<h3>Async</h3>
LLM Engine scales async endpoints based on the task queue length. Autoscaling based on external queues is not a standard feature in K8s, so LLM Engine contains an Autoscaler component. The Autoscaler polls queues for their lengths and then adjusts the workers for corresponding endpoints. The autoscaler simply updates the replicas field for the corresponding k8s deployments objects. The desired replicas value is calculated as the &ldquo;max queue length over the last 10 minutes&rdquo; divided by the user provided requests_per_worker value.
<h3>Scale From Zero</h3>
Some endpoints benefit from scaling down to 0 workers for particular use cases. An example would be experimental endpoints only used for demos or model zoo endpoints that receive very little traffic. To support scaling from zero, LLM Engine makes use of <a href="https://keda.sh/" target="_blank" rel="noopener">KEDA</a>. A KEDA scaler is configured to scale on the length of a Redis list. Inside the Gateway, LLM Engine pushes items to the Redis list, but caps the length of the list at one, in addition to setting an expiration time on the list. This has the effect of enabling scaling from zero pods to one pod. The request is saved inside the Gateway so that LLM Engine can wait for the pod to scale up to serve the request.
<h3>Cacher</h3>
For the request volume LLM Engine makes to the K8s API, the K8s API server is slow and easily overwhelmed. Simple list pod operations can take over a few seconds, especially as the number of pods grows in the cluster. LLM Engine has a cacher service that periodically queries for the Kubernetes API for all relevant cluster states and saves it to Redis cache. This takes load off the core K8s API server and significantly improves latency for simple GET requests sent by users.
The same service is responsible for updating Endpoint image caches in the nodes. This enables different endpoints to autoscale much faster, enabling users to efficiently provision GPU resources for their workloads. Without fast autoscaling, users would often bump up their min_workers to a value much higher than required and consequently rack up a higher compute bill. For details about fast autoscaling, we defer to a <a href="https://keda.sh/" target="_blank" rel="noopener">previous blog post</a> dedicated to this topic.
<h3>Self-Hosting</h3>
LLM Engine's self-hosting feature is compatible with AWS and Azure environments. It utilizes a Helm chart for configuring and managing the LLM Server, ensuring seamless integration within a cloud environment. Key components of this setup (visualized in our system diagram above) include dependencies like PostgreSQL for database needs and Redis for caching and queuing.
Overall, this self-hosting setup offers a robust, scalable solution for deploying LLM Engine, aligning with contemporary cloud infrastructure needs and providing a flexible, powerful environment for LLM applications.
Get started by downloading the <a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener">open-source repo from Github</a>. If you need a full-stack platform to customize, build, test, and deploy enterprise-ready Generative AI models and production applications, you can learn more about <a href="https://scale.com/genai-platform">Scale GenAI Platform</a>.
<h3 dir="ltr">Acknowledgments</h3>
Special thanks to the Scale team that made this work possible:
Sean Shi, Yi Xu, Will Song, Yunfeng Bai, Katie Wu, Sai Atmakuri, Ian Macleod

Learn how Scale optimizes Open-Source model hosting with LLM Engine, available as an Open-Source package and as part of Scale GenAI Platform.

Blog