Skip to content

proxystore.endpoint.files

Helpers for files in endpoint directories.

VersionedFile

Bases: BaseModel

Base model of a file in an endpoint directory.

Each file written by ProxyStore has a format version which is incremented on incompatible changes to the file format. Subclasses set the default of version to the version supported by this version of ProxyStore, and a file with any other version is rejected.

Example 
class MyFile(VersionedFile):
    DESCRIPTION: ClassVar[str] = 'my file'

    version: int = 1
    value: str

Attributes:

  • DESCRIPTION (str) –

    Description of the file for error messages.

  • version (int) –

    Format version of the file.

write_private_file 

write_private_file(path: str, data: BytesLike) -> None

Atomically write data to a file that only the owner can access.

The data is written to a temporary file with mode 0600 in the same directory which then replaces path, so readers never observe a partially written file.

Source code in proxystore/endpoint/files.py 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
def write_private_file(path: str, data: BytesLike) -> None:
    """Atomically write data to a file that only the owner can access.

    The data is written to a temporary file with mode `0600` in the same
    directory which then replaces `path`, so readers never observe a
    partially written file.
    """
    fd, tmp_path = tempfile.mkstemp(
        dir=os.path.dirname(path) or '.',
        prefix=f'.{os.path.basename(path)}.',
    )
    try:
        with os.fdopen(fd, 'wb') as f:
            f.write(data)
        os.replace(tmp_path, path)
    except BaseException:
        with contextlib.suppress(FileNotFoundError):
            os.remove(tmp_path)
        raise

read_model 

read_model(model: type[FileT], path: str) -> FileT

Read a file written by an endpoint.

The file is parsed as TOML if path ends with .toml and as JSON otherwise.

Raises:

Source code in proxystore/endpoint/files.py 
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
def read_model(model: type[FileT], path: str) -> FileT:
    """Read a file written by an endpoint.

    The file is parsed as TOML if `path` ends with `.toml` and as JSON
    otherwise.

    Raises:
        FileNotFoundError: If the file does not exist.
        EndpointConfigError: If the file is malformed or has an unsupported
            format version.
    """
    with open(path, 'rb') as f:
        contents = f.read()
    try:
        if path.endswith('.toml'):
            return model.model_validate(
                tomllib.loads(contents.decode()),
                strict=True,
            )
        return model.model_validate_json(contents, strict=True)
    except ValueError as e:
        # Includes decoding and pydantic validation errors.
        raise EndpointConfigError(
            f'The {model.DESCRIPTION} at {path} is malformed: {e}',
        ) from None

write_model 

write_model(path: str, instance: VersionedFile) -> None

Atomically write a file only the owner can access.

The file is written as TOML if path ends with .toml and as JSON otherwise. See write_private_file().

Source code in proxystore/endpoint/files.py 
113
114
115
116
117
118
119
120
121
122
123
124
def write_model(path: str, instance: VersionedFile) -> None:
    """Atomically write a file only the owner can access.

    The file is written as TOML if `path` ends with `.toml` and as JSON
    otherwise. See
    [`write_private_file()`][proxystore.endpoint.files.write_private_file].
    """
    if path.endswith('.toml'):
        data = dumps(instance)
    else:
        data = instance.model_dump_json(indent=2)
    write_private_file(path, data.encode())