Connector

The Connector is a Protocol that defines the low-level interface to a mediated communication channel or object store. The Connector methods operate on bytes-like data and keys which are tuples of metadata that can identify a unique object.

The protocol is as follows:

KeyT = TypeVar('KeyT', bound=NamedTuple)


class Connector(Protocol[KeyT]):
    def close(self, *, clear: bool | None = None) -> bool | None: ...
    def config(self) -> dict[str, Any]: ...
    @classmethod
    def from_config(cls, config: dict[str, Any]) -> Self: ...
    def evict(self, key: KeyT) -> None: ...
    def exists(self, key: KeyT) -> bool: ...
    def get(self, key: KeyT) -> BytesLike | None: ...
    def get_batch(self, keys: Sequence[KeyT]) -> list[BytesLike | None]: ...
    def put(self, obj: BytesLike) -> KeyT: ...
    def put_batch(self, objs: Sequence[BytesLike]) -> list[KeyT]: ...

Implementations

Implementing a custom Connector requires creating a class which implements the above methods. Note that the custom class does not need to inherit from Connector because it is a Protocol.

Many Connector implementations are provided in the proxystore.connectors module, and users can easily create their own. A Connector instance is used by the Store to store and retrieve serialized objects.

The clear argument of close() controls whether the objects stored by the connector are also removed (e.g., deleting the directory used by the FileConnector). When clear is None, the connector should use its own default. close() should return True if the objects were removed, so the owner Store knows its proxies can no longer be resolved in this process. A Store only clears its connector when closed if it is the owner; stores created implicitly, such as when a proxy is resolved in another process, always pass clear=False.

Connectors must be thread-safe. A Store may call the methods of one connector from multiple threads at the same time, including with the same key (e.g., two threads evicting a key while another gets it). If a connector uses a client which is not thread-safe, guard the client with a lock.

Implementations of put() and put_batch() may accept additional, connector-specific keyword arguments. These are passed from Store methods via the connector_options parameter. For example, the MultiConnector routes objects based on tags.

store.put(obj, connector_options={'subset_tags': ['small']})

Extensions

A Connector implementation can be extended to implement the DeferrableConnector protocol. A DeferrableConnector provides methods for creating a key and then setting that key to an object at a later time. Not all of the provided Connector implementations implement the DeferrableConnector protocol because some transfer methods require the object before creating a key for that object.

Security

A Store deserializes the bytes a connector returns with pickle by default, so anyone who can write to a connector's storage can run code in the processes that read from it (see Store Security). Connectors differ in who can read and write their objects.

• LocalConnector: only the current process.
• FileConnector: anyone with access to the directory. If the connector creates the directory, only you can access it and its files by default (dir_mode=0o700, file_mode=0o600). Set dir_mode=0o770, file_mode=0o660 to share objects with a group. A directory that already exists keeps its permissions, so check them before using an existing directory on a shared system.
• RedisConnector: anyone who can connect to the Redis server, as set by the server's password or access control lists. Pass username, password, and ssl=True (with the other ssl_* options) in client_options to authenticate and encrypt connections. The client_options are kept in the connector's configuration, so a password is included in every proxy and in a StoreConfig written to a file.
• ZeroMQConnector: anyone who can reach the server's port. The servers do not authenticate clients or encrypt data, so only use this connector within a trusted network, such as the interconnect of an HPC cluster.
• EndpointConnector: processes that can read the endpoint directory, and peer endpoints you allow. See Endpoint Security.
• GlobusConnector: anyone with access to the directories on the Globus collections, as set by the file systems and the permissions of each collection.
• DAOSConnector: anyone with access to the DAOS container, as set by the DAOS pool and container access control lists.
• MultiConnector: anyone who can access any of its connectors.