nip.language_model_server.types.VllmStartRequest

Contents

nip.language_model_server.types.VllmStartRequest#

class nip.language_model_server.types.VllmStartRequest(*, model_name: str, quantization: Literal['bitsandbytes', 'none'] = 'none')[source]#

A request to start the vLLM server with a specific model.

Attributes

__fields_set__

model_computed_fields

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_extra

Get extra fields set during validation.

model_fields

model_fields_set

Returns the set of fields that have been explicitly set on this model instance.

model_name

The name of the model to be served by the vLLM server.

quantization

The quantization method to use for the model.

Methods