*********************
Distributed Computing
*********************

Sage comes built in with a powerful distributed computing framework
called Distributed Sage (``dsage``).

Overview
========

Distributed Sage is a framework that allows one to do distributed
computing from within Sage. It includes a server, client and
workers as well as a set of classes that one can subclass from to
write distributed computation jobs. It is designed to be used
mainly for 'coarsely' distributed computations, i.e., computations
where jobs do not have to communicate much with each other. This is
also sometimes referred to as 'grid' computing.

There are 3 parts that make up Distributed Sage:


#. The **server** is responsible for job distribution, submission
   and collection. It also includes a web interface from which you can
   monitor your jobs and do other administrative tasks.

#. The **client** is responsible for submitting new jobs to the
   server and collecting the results.

#. The **workers** perform the actual computations.


Quick Start
===========

Here are a few illustrations of how get up and running with
``dsage``.

Example 1
---------


#. Run ``dsage.setup()``. This will setup the SQLite database and
   generate a private and public key to be used for SSL communication.
   It will also add a default user whose username defaults to your
   current username.

#. Run ``d = dsage.start_all()``. This command will launch the
   server, the web server, :math:`2` workers and return an object
   (``d``) which is a connection to the server. From here on, your
   interaction with ``dsage`` will be mainly though the ``d`` object.

#. Open up your browser and go to http://localhost:8082 to see the
   web interface of ``dsage``. From here you will be able to see the
   status of your jobs, the workers connected and other important
   information about your ``dsage`` server.

#. Let's begin with a simple example. Type ``job = d('2+2')``. If you
   look at the web interface, you should see that there is a new job
   in the table. Now one of your workers will fetch that job, execute
   it and present you the result. To get at the result, type
   ``job.result``. It might not be there yet because for this simple
   computation, the network communication overhead dominates the
   computational time. If you want to wait for your job to finish, you
   can call ``job.wait()`` which will block until the job completes, at
   which time you can inspect ``job.result`` for the result. You can do
   any computation in this way by calling ``d``.


Example 2
---------

In this example, we will show you how to use the ``DistributedFactor``
class that comes built-in with ``dsage``. DistributedFactor attempts
to factor numbers by using a combination of the ECM and the QSieve
algorithm, as well as trial factorization for small factors.


#. Run ``d = dsage.start_all()`` if you have not started your
   ``dsage`` session yet, otherwise you can continue to use the previous
   ``d`` instance.

#. Start the distributed factoring job with
   ``factor_job = DistributedFactor(d, number)``. You can pick fairly
   large values, try for example :math:`2^{360}-1`. You can see
   whether or not the factoring job is done by looking at the
   ``factor_job.done`` attribute. When it is done, you can look at the
   prime factors it found by inspecting
   ``factor_job.prime_factors``.

Example 3
---------

This example demonstrates a distributed version of the ``map``
method. You can find the documentation for the regular ``map`` method
here: http://docs.python.org/lib/built-in-funcs.html. The syntax is
exactly the same.

First, run ``d = dsage.start_all()`` if you have not started your
``dsage`` session yet; otherwise, you can continue to use the previous
``d`` instance.

.. skip

::

    sage: def f(n): return n*n
    sage: j = d.map(f, [25,12,25,32,12])
    sage: jobs = d.block_on_jobs(j)

This will block until ``f`` has been evaluated for each of the inputs.

Example 4
---------

This example demonstrates how to use the ``@parallel`` decorator with
a ``dsage`` instance.

First, run ``d = dsage.start_all()`` if you have not started your
``dsage`` session yet; otherwise, you can continue to use the previous
``d`` instance.

.. skip

::

    sage: P = parallel(p_iter = d.parallel_iter)
    sage: @P
    ...   def f(n,m):  return n+m
    sage: f([(1,2), (5, 10/3)])



Files
=====

``dsage`` stores a few files in ``$SAGE_ROOT/.sage/dsage``:


#. ``pubcert.pem`` and ``cacert.pem``: the public and private keys the
   server uses for SSL.

#. ``dsage_key.pub`` and ``dsage_key``: the keys used for
   authenticating the user.

#. The directory ``db/``: this contains the ``dsage`` database.

#. ``*.log`` files: logs generated by the server and workers.

#. the directory ``tmp_worker_files/``: workers store the jobs they
   have processed here.

