ProxyStore

ProxyStore provides pass-by-reference semantics for distributed Python applications via transparent object proxies.

A proxy is a lightweight reference to an object in remote storage that can be cheaply sent to any process, even on a remote machine. The proxy resolves its target object just-in-time when first used and then behaves like the target object, so code consuming a proxy needs no changes and no knowledge of how the data is stored or moved. This reduces transfer overheads through intermediaries, such as workflow schedulers or cloud services, and decouples application logic from communication code.

ProxyStore is used to build:

• Task-based workflows (e.g., Dask Distributed)
• Serverless applications (e.g., Globus Compute)
• Distributed futures
• Bulk data streaming

Objects can be stored in and transferred via shared file systems, Redis, ZeroMQ, Globus Transfer, or ProxyStore Endpoints for peer-to-peer transfer. See the Connectors reference for all options or to implement your own.

Learn more in the Concepts overview and the complete documentation at docs.proxystore.dev.

Installation

The base ProxyStore package can be installed with pip.

pip install proxystore

Leveraging third-party libraries may require dependencies not installed by default but can be enabled via extras installation options (e.g., endpoints, kafka, or redis). All additional dependencies can be installed with:

pip install proxystore[all]

See the Installation guide for more information about the available extras installation options. See the Contributing guide to get started for local development.

Example

Proxies are cheap to send to other processes and resolve themselves when used.

from concurrent.futures import ProcessPoolExecutor

from proxystore.connectors.file import FileConnector
from proxystore.store import Store


def process(data: dict[str, str]) -> str:
    # The proxy resolves itself to the dict when first used and
    # then behaves exactly like the dict.
    return data['hello']


if __name__ == '__main__':
    with Store('example', FileConnector('./proxystore-data')) as store:
        # Put the object in the store and get back a proxy, a lightweight
        # reference which is cheap to send to other processes.
        proxy = store.proxy({'hello': 'world'})

        # Functions can be invoked with the proxy without any changes.
        with ProcessPoolExecutor() as pool:
            assert pool.submit(process, proxy).result() == 'world'

Check out the Get Started guide to learn more!

Citation

If you use ProxyStore or any of this code in your work, please cite our ProxyStore (SC '23) and Proxy Patterns (TPDS) papers.

@inproceedings{pauloski2023proxystore,
    title = {Accelerating {C}ommunications in {F}ederated {A}pplications with {T}ransparent {O}bject {P}roxies},
    author = {Pauloski, J. Gregory and Hayot-Sasson, Valerie and Ward, Logan and Hudson, Nathaniel and Sabino, Charlie and Baughman, Matt and Chard, Kyle and Foster, Ian},
    address = {New York, NY, USA},
    articleno = {59},
    booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
    doi = {10.1145/3581784.3607047},
    isbn = {9798400701092},
    location = {Denver, CO, USA},
    numpages = {15},
    publisher = {Association for Computing Machinery},
    series = {SC '23},
    url = {https://doi.org/10.1145/3581784.3607047},
    year = {2023}
}

@article{pauloski2024proxystore,
    title = {Object {P}roxy {P}atterns for {A}ccelerating {D}istributed {A}pplications},
    author = {Pauloski, J. Gregory and Hayot-Sasson, Valerie and Ward, Logan and Brace, Alexander and Bauer, André and Chard, Kyle and Foster, Ian},
    doi = {10.1109/TPDS.2024.3511347},
    journal = {IEEE Transactions on Parallel and Distributed Systems},
    number = {},
    pages = {1-13},
    volume = {},
    year = {2024}
}