Distributed Computing

Sage comes built in with a powerful distributed computing framework called Distributed Sage (dsage).

Overview

Distributed Sage is a framework that allows one to do distributed computing from within Sage. It includes a server, client and workers as well as a set of classes that one can subclass from to write distributed computation jobs. It is designed to be used mainly for ‘coarsely’ distributed computations, i.e., computations where jobs do not have to communicate much with each other. This is also sometimes referred to as ‘grid’ computing.

There are 3 parts that make up Distributed Sage:

1. The server is responsible for job distribution, submission and collection. It also includes a web interface from which you can monitor your jobs and do other administrative tasks.
2. The client is responsible for submitting new jobs to the server and collecting the results.
3. The workers perform the actual computations.

Quick Start

Here are a few illustrations of how get up and running with dsage.

Example 1

1. Run dsage.setup(). This will setup the SQLite database and generate a private and public key to be used for SSL communication. It will also add a default user whose username defaults to your current username.
2. Run d = dsage.start_all(). This command will launch the server, the web server, workers and return an object (d) which is a connection to the server. From here on, your interaction with dsage will be mainly though the d object.
3. Open up your browser and go to http://localhost:8082 to see the web interface of dsage. From here you will be able to see the status of your jobs, the workers connected and other important information about your dsage server.
4. Let’s begin with a simple example. Type job = d('2+2'). If you look at the web interface, you should see that there is a new job in the table. Now one of your workers will fetch that job, execute it and present you the result. To get at the result, type job.result. It might not be there yet because for this simple computation, the network communication overhead dominates the computational time. If you want to wait for your job to finish, you can call job.wait() which will block until the job completes, at which time you can inspect job.result for the result. You can do any computation in this way by calling d.

Example 2

In this example, we will show you how to use the DistributedFactor class that comes built-in with dsage. DistributedFactor attempts to factor numbers by using a combination of the ECM and the QSieve algorithm, as well as trial factorization for small factors.

1. Run d = dsage.start_all() if you have not started your dsage session yet, otherwise you can continue to use the previous d instance.
2. Start the distributed factoring job with factor_job = DistributedFactor(d, number). You can pick fairly large values, try for example . You can see whether or not the factoring job is done by looking at the factor_job.done attribute. When it is done, you can look at the prime factors it found by inspecting factor_job.prime_factors.

Example 3

This example demonstrates a distributed version of the map method. You can find the documentation for the regular map method here: http://docs.python.org/lib/built-in-funcs.html. The syntax is exactly the same.

First, run d = dsage.start_all() if you have not started your dsage session yet; otherwise, you can continue to use the previous d instance.

sage: def f(n): return n*n
sage: j = d.map(f, [25,12,25,32,12])
sage: jobs = d.block_on_jobs(j)

This will block until f has been evaluated for each of the inputs.

Example 4

This example demonstrates how to use the @parallel decorator with a dsage instance.

First, run d = dsage.start_all() if you have not started your dsage session yet; otherwise, you can continue to use the previous d instance.

sage: P = parallel(p_iter = d.parallel_iter)
sage: @P
...   def f(n,m):  return n+m
sage: f([(1,2), (5, 10/3)])

Files

dsage stores a few files in $SAGE_ROOT/.sage/dsage:

1. pubcert.pem and cacert.pem: the public and private keys the server uses for SSL.
2. dsage_key.pub and dsage_key: the keys used for authenticating the user.
3. The directory db/: this contains the dsage database.
4. *.log files: logs generated by the server and workers.
5. the directory tmp_worker_files/: workers store the jobs they have processed here.