/usr/share/doc/python-distributed-doc/html/_sources/limitations.rst.txt is in python-distributed-doc 1.20.2+ds.1-2.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | Limitations
===========
Dask.distributed has limitations. Understanding these can help you to reliably
create efficient distributed computations.
Performance
-----------
- The central scheduler spends a few hundred microseconds on every task. For
optimal performance, task durations should be greater than 10-100ms.
- Dask can not parallelize within individual tasks. Individual tasks should
be a comfortable size so as not to overwhelm any particular worker.
- Dask assigns tasks to workers heuristically. It *usually* makes the right
decision, but non-optimal situations do occur.
- The workers are just Python processes, and inherit all capabilities and
limitations of Python. They do not bound or limit themselves in any way.
In production you may wish to run dask-workers within containers.
Assumptions on Functions and Data
---------------------------------
Dask assumes the following about your functions and your data:
- All functions must be serializable either with pickle or
`cloudpickle <https://github.com/cloudpipe/cloudpickle>`_. This is
*usually* the case except in fairly exotic situations. The
following should work::
from cloudpickle import dumps, loads
loads(dumps(my_object))
- All data must be serializable either with pickle, cloudpickle, or using
Dask's custom serialization system.
- Dask may run your functions multiple times,
such as if a worker holding an intermediate result dies. Any side effects
should be `idempotent <https://en.wikipedia.org/wiki/Idempotence>`_.
-
Security
--------
As a distributed computing framework, Dask enables the remote execution of
arbitrary code. You should only host dask-workers within networks that you
trust. This is standard among distributed computing frameworks, but is worth
repeating.
|