eBay’s new internal big language model for e-commerce can also translate

In a June 17, 2024 article, eBay introduced its own series of Large Language Models (LLMs), tailored specifically for the e-commerce sector.

Named LiLiuM 1B, 7B and 13B, these models have been developed in-house to meet eBay’s specific needs for a variety of applications, including translation, in the e-commerce domain, providing full control over licensing, data, vocabulary and architecture.

The authors stated that “these models are intended to eliminate the dependence on external LLMs on eBay.”

They explained that using base models that can be accessed and tuned for specific use cases, such as the LLaMA-2 models, “poses risks in terms of licensing, data security and future-proofing, among others.” They further noted that “these models are very general and mostly trained on English-centric data.”

They developed LLM completely in-house from scratch, using a massive 3 trillion token dataset, covering both general texts and e-commerce specific texts in multiple languages. They used the ParaCrawl corpus along with a smaller internal corpus from the e-commerce domain. This approach gives them the robustness to support various languages ​​and domain-specific tasks.

In addition, eBay has developed its own tokenizer and model vocabulary, tailored to e-commerce needs. “This gives us several benefits, namely (i) full control over the vocabulary, including special tokens, (ii) better support for multilingualism and (iii) better customization for specific e-commerce use cases,” they said.

Slator 2024 Language Industry Market Report – AI Language Editing

The 140-page flagship report provides in-depth analysis of the market, language AI capabilities, survey results and much more.

Eliminating dependencies

According to the authors, their models perform comparably or better than the popular LLaMA-2 models, particularly excelling at machine translations into languages ​​other than English, as well as natural language understanding (NLU) tasks and e-commerce-specific applications.

The authors explained that this performance boost is attributed to the inclusion of significant amounts of non-English and e-commerce-specific data during pre-training, which improves the models’ understanding and performance on non-English tasks. Additionally, the tailored vocabulary for e-commerce tasks resulted in significant speedups in text generation, outperforming LLaMA-2 by up to 34%.

The authors expect these models “to be used as a basis for tuning and fine-tuning instructions, eliminating dependencies on external models.”

Future work will focus on improving the data flow, incorporating more eBay-specific data, training larger models, and exploring mixed-expertise architectures to improve performance.

Authors: Christian Herold, Michael Kozielski, Leonid Ekimov, Pavel Petrushkov, Pierre-Yves Vandenbussche and Shahram Khadivi