Data Mobility Exhibition

The Globus endpoints and test data sets listed below are available for use by participants in the Data Mobility Exhibition. Thank you to the host institutions that that have generously made these available to us!

Usage Guidelines and Ground Rules

1. Data sets are stored either in the endpoint root or /datasets directory. You have read-only access to this directory so you can initiate outbound transfers to your own destination endpoint.
2. If you wish to use one of these the endpoints as the destination, please write files to a sub-directory within the root or /perftest directory; use a descriptive name for your subdirectory so that it's easily identifiable by all, e.g. esnet-team.
3. Please be careful to not overwrite or otherwise tamper with another team's subdirectory.
4. All participants in the exhibition are sharing the same storage allocation, so please be sure to delete all transferred files as soon as your transfer completes.
5. While most endpoints have very large/open quotas, some may be limited as shown. Do not attempt to transfer a data set that is larger than the quota. Also, please monitor your transfers (particularly those using the 5TB data set), and be prepared to cancel and immediately clean up any transferred files if you bump up against space limits.

Thanks for being good data mobility citizens!

Data Sets

Name	Total Size	Description
ds01	100MB	10,000 x 10KB files in single directory
ds04	10GB	10,000 x 1MB files in 100 non-nested directories, 100 files/directory
ds06	100GB	100,000 x 1MB files in single directory
ds08	1TB	50 x 10GB; 350 x 1GB; 1,000 x 100MB; 5,500 x 10MB; 23,176 x 1MB files in single directory
ds10	1TB	100 x 10GB files in single directory
ds12	100GB	1 x 100GB file in single directory
ds14	5TB	50 x 100GB files in single directory
ds16	1TB	4 x 250GB files in single directory

Endpoints

The following endpoints contain a copy of each data set; they are read-only, so you may use them as the source endpoint for your transfers. Endpoint IDs are in parentheses, in case you are scripting or otherwise automating your transfers.

Display Name	Description
cac_dtn_test (606579ae-5b03-11e9-bf32-0edbf3a4e7ee)	Cornell Center for Advanced Computing in Ithaca, NY
DME Datasets - Boston (e6992bda-c0b4-11e9-98dc-0a63aa6b37da)	Lustre filesystem located in Harvard's Boston datacenter
DME Datasets - MGHPCC (6e8e1a56-beb6-11e9-9ceb-0edb67dd7a14)	Lustre filesystem located at the MGHPCC
DME Datasets - Argonne (e261ffb8-6d04-11e5-ba46-22000b92c6ec)	IBM Spectrum Scale filesystem on the Petrel system at Argonne National Lab
Pamela Hill Data Share (924a32b0-6a2a-11e6-83a8-22000b97daec)	NCAR GLADE (IBM Spectrum Scale) filesystem in Boulder, CO

The following endpoints are write-enabled so you may use them as the destination endpoint for your transfers. Click on the Display Name link to open the endpoint in the Globus web app file manager.

Display Name	Description	Quota
cac_dtn_test (606579ae-5b03-11e9-bf32-0edbf3a4e7ee)	Cornell Center for Advanced Computing in Ithaca, NY	Unknown
DME PerfTest - Boston (38b0ee04-c0b4-11e9-939a-02ff96a5aa76)	Lustre filesystem located in Harvard's Boston datacenter	10TB
DME PerfTest - MGHPCC (8c799234-be52-11e9-9ceb-0edb67dd7a14)	Lustre filesystem located at the MGHPCC in Holyoke, MA	Unknown
DME PerfTest - Argonne (e261ffb8-6d04-11e5-ba46-22000b92c6ec)	IBM Spectrum Scale filesystem on the Petrel system at Argonne National Lab (Petrel detailed description)	10TB
DME PerfTest - Michigan (0ddccd16-c04a-11e9-9cec-0edb67dd7a14)	Turbo system on the Flux cluster at the University of Michigan	100GB
Pamela Hill Data Share (924a32b0-6a2a-11e6-83a8-22000b97daec)	NCAR GLADE (IBM Spectrum Scale) filesystem in Boulder, CO This endpoint has multiple 40Gbps links to the Internet2 100Gbps backbone so you should see great performance (assuming the destination endpoint is well-provisioned).	Unknown
DME PerfTest - Google Drive (fc185009-345f-4a62-9ae3-21e8e79960c6)	Google Drive cloud storage accessed via an EC2 DTN in the us-east-1 region Note: Google Drive imposes rate limits on their API so expect throttling when transferring datasets with many files.	None
DME PerfTest - Box (401f8d2d-8797-4611-8521-196870dbf9fd)	Box cloud accessed via an EC2 DTN in the us-east-1 region Note: In addition to API rate limits, Box imposes a filesize limit of 15GB; do not attempt to transfer ds12, ds14, or ds16.	None

Useful Links

• The Globus Connect Server installation guide contains instructions for setting up your Globus endpoint.
  Note: Unless you are a paying Globus subscriber at the High Assurance level or above, you must follow the instructions for Globus Connect Server v4; Globus Connect Server v5 is not the latest, general purpose release.
• The Globus Command Line Interface (CLI) is great for scripting a set of transfers that you can run automatically from multiple endpoints.
• Additional (and more complete) examples of scripts are available in the Globus GitHub repositories; in particular, this repo has bash and Python equivalents for common automation tasks.
• Here are some (many!) of the frequently asked questions about Globus.
• The Globus support team is always available to help; please contact support@globus.org.

Click here for more documentation and other helpful Globus resources.