Videospeicherverwaltungsverfahren für Grosssprachenmodellinferenz, Vorrichtungen, Medien und Produkte

Details

Veröffentlichungs-Nr.

EP4730142

Anmeldetag

11. Dezember 2024

Veröffentlichung

22. April 2026

Rechtsraum

EP

Abstract

This specification provides video memory management methods for large language model inference, devices, media, and products, which are applied to a service device deployed with a large language model. The method includes: allocating physical video memory resources on the service device, to separately map the physical video memory resources to a first video memory resource pool in which a cache object is a key-value cache and a second video memory resource pool in which a cache object is an intermediate activation value; and for an inference task submitted to the large language model, upon determining that an idle video memory resource in any video memory resource pool is insufficient to cache a corresponding cache object for the inference task, temporarily transferring at least a portion of idle video memory resources in another video memory resource pool to the any video memory resource pool.

Anmelder

Firma

Alipay (Hangzhou) Information Technology Co., Ltd.

Land

🇨🇳 China