Data Mobility Exhibition

The Globus endpoints and test data sets listed below are available for use by participants in the Data Mobility Exhibition. Thank you to the host institutions that that have generously made these available to us!

Usage Guidelines and Ground Rules

  1. Data sets are stored either in the endpoint root or /datasets directory. You have read-only access to this directory so you can initiate outbound transfers to your own destination endpoint.
  2. If you wish to use one of these the endpoints as the destination, please write files to a sub-directory within the root or /perftest directory; use a descriptive name for your subdirectory so that it's easily identifiable by all, e.g. esnet-team.
  3. Please be careful to not overwrite or otherwise tamper with another team's subdirectory.
  4. All participants in the exhibition are sharing the same storage allocation, so please be sure to delete all transferred files as soon as your transfer completes.
  5. While most endpoints have very large/open quotas, some may be limited as shown. Do not attempt to transfer a data set that is larger than the quota. Also, please monitor your transfers (particularly those using the 5TB data set), and be prepared to cancel and immediately clean up any transferred files if you bump up against space limits.
Thanks for being good data mobility citizens!

Data Sets

Name Total Size Description
ds01100MB10,000 x 10KB files in single directory
ds0410GB10,000 x 1MB files in 100 non-nested directories, 100 files/directory
ds06100GB100,000 x 1MB files in single directory
ds081TB50 x 10GB; 350 x 1GB; 1,000 x 100MB; 5,500 x 10MB; 23,176 x 1MB files in single directory
ds101TB100 x 10GB files in single directory
ds12100GB1 x 100GB file in single directory
ds145TB50 x 100GB files in single directory
ds161TB4 x 250GB files in single directory

Endpoints

The following endpoints contain a copy of each data set; they are read-only, so you may use them as the source endpoint for your transfers. Endpoint IDs are in parentheses, in case you are scripting or otherwise automating your transfers.

Important Note!

You must be a member of the "DME Endpoint Access" Globus group to access the endpoints.
click here to request membership
Display Name Description
cac_dtn_test
(606579ae-5b03-11e9-bf32-0edbf3a4e7ee)
Cornell Center for Advanced Computing in Ithaca, NY
DME Datasets - Boston
(e6992bda-c0b4-11e9-98dc-0a63aa6b37da)
Lustre filesystem located in Harvard's Boston datacenter
DME Datasets - MGHPCC
(6e8e1a56-beb6-11e9-9ceb-0edb67dd7a14)
Lustre filesystem located at the MGHPCC
DME Datasets - Argonne
(e261ffb8-6d04-11e5-ba46-22000b92c6ec)
IBM Spectrum Scale filesystem on the Petrel system at Argonne National Lab
Pamela Hill Data Share
(924a32b0-6a2a-11e6-83a8-22000b97daec)
NCAR GLADE (IBM Spectrum Scale) filesystem in Boulder, CO
The following endpoints are write-enabled so you may use them as the destination endpoint for your transfers. Click on the Display Name link to open the endpoint in the Globus web app file manager.

Display Name Description Quota
cac_dtn_test
(606579ae-5b03-11e9-bf32-0edbf3a4e7ee)
Cornell Center for Advanced Computing in Ithaca, NY Unknown
DME PerfTest - Boston
(38b0ee04-c0b4-11e9-939a-02ff96a5aa76)
Lustre filesystem located in Harvard's Boston datacenter 10TB
DME PerfTest - MGHPCC
(8c799234-be52-11e9-9ceb-0edb67dd7a14)
Lustre filesystem located at the MGHPCC in Holyoke, MA Unknown
DME PerfTest - Argonne
(e261ffb8-6d04-11e5-ba46-22000b92c6ec)
IBM Spectrum Scale filesystem on the Petrel system at Argonne National Lab (Petrel detailed description) 10TB
DME PerfTest - Michigan
(0ddccd16-c04a-11e9-9cec-0edb67dd7a14)
Turbo system on the Flux cluster at the University of Michigan 100GB
Pamela Hill Data Share
(924a32b0-6a2a-11e6-83a8-22000b97daec)
NCAR GLADE (IBM Spectrum Scale) filesystem in Boulder, CO
This endpoint has multiple 40Gbps links to the Internet2 100Gbps backbone so you should see great performance (assuming the destination endpoint is well-provisioned).
Unknown
DME PerfTest - Google Drive
(fc185009-345f-4a62-9ae3-21e8e79960c6)
Google Drive cloud storage accessed via an EC2 DTN in the us-east-1 region
Note: Google Drive imposes rate limits on their API so expect throttling when transferring datasets with many files.
None
DME PerfTest - Box
(401f8d2d-8797-4611-8521-196870dbf9fd)
Box cloud accessed via an EC2 DTN in the us-east-1 region
Note: In addition to API rate limits, Box imposes a filesize limit of 15GB; do not attempt to transfer ds12, ds14, or ds16.
None

Useful Links

Click here for more documentation and other helpful Globus resources.