GlobusWorld 2025 Program

This presentation details Northwestern University's implementation of Globus Connect Server (GCS) on Amazon Web Services (AWS) to facilitate the migration of a research data archive from on-premises storage to AWS S3. We will walk through the complete deployment process, including prerequisites, configuration requirements, and architectural decisions for both the on-premises and AWS environments. The talk examines key aspects of security, networking, and identity management, while sharing practical insights on performance optimization and scalability considerations encountered during the implementation. We discuss operational approaches, cost considerations for different usage patterns, and lessons learned from our deployment experience. To help other institutions undertaking similar projects, we will share our AWS CloudFormation template and configuration guidelines for GCS deployment. This technical deep-dive will benefit system administrators and research computing professionals planning to leverage Globus for cloud data transfer and management.

We also introduce The S3 Connection Wizard: Globus, an easy-to-use tool that transforms how researchers share and access data across institutions using the AWS S3 Connector for Globus. By simplifying a manual multi-step process to an intuitive wizard interface, this tool enables researchers to more easily leverage the combination of Globus' data sharing capabilities and AWS cloud storage.

We have developed a solution for cold data management using Globus, with a Globus Flow as the primary interface. For large datasets, users can invoke a Globus Flow that calls an open-source tool, suitcasectl. This tool chunks data into TAR or ZIP files and generates an inventory with associated metadata. All data is stored as S3 objects, with inventories being separately searchable.

For more information, visit: sites.duke.edu/fastresearchstorage/suitcasectl.

The Rosalind Franklin Institute has been using Globus for 3 years to transfer high-resolution multimodal biological data from our instruments to our analysis platforms. Approximately Franklin is generating around 70 terabytes a month, which requires high speed and robust transfers. Distributed data transfer solutions, such as Globus can facilitate this goal, however optimally they should be used in automated pipelines that do not require a human in the loop.

We have been working towards this in 2 ways. Firstly, we have developed and released FlowCron [1] which is a Function-as-a-Service software tool that can facilitate users to access HPCs to process their data using a Globus Flow to accommodate any data transfers. This significantly reduces the time to science.

Secondly, we would like to present our next step with release of Rosalind Franklin Institute’s GlobusAPI. Our RFI-GlobusAPI container image provides an intuitive interface with Globus Python SDK, enabling the easy deployment of Globus microservices, which facilitates the incorporation of Globus within automated data processing pipelines. RFI-GlobusAPI offers multiple useful high in abstraction commands, and it has already been tested as part of ArgoWF workflows. Furthermore, RFI-GlobusAPI is open source and available to all. Our future goals include the expansion of RFI-GlobusAPI with further utility, allowing us to incorporate it into different parts of our data lifecycle to create a secure, scalable, reproducing and efficient automated data infrastructure that can provide FAIR [2,3] data for all our science.

[1] doi.org/10.12688/wellcomeopenres.23491.1
[2] doi.org/10.1038/sdata.2016.18
[3] www.go-fair.org/fair-principles

During this talk, I'll go over the motivations and progress around deploying Globus Compute Multi-Users Endpoints on SDSC's Expanse system and the realities of adding new functionality to an existing system as new systems are being deployed. Just like we've done with Globus Connect Server, having ubiquitous deployments of Globus Compute Multi-User could provide a uniform mechanism for programmatic access to all of SDSC's HPC resources. This has several in terms of security, access modalities, integration with other systems, and automation. At present, science gateways typically submit jobs to HPC systems using SSH keys without passphrases. This is a known risk and mitigating controls are used to reduce the impact if the private key is stolen. Regular users have no programmatic access and must use MFA when authenticating via SSH. Deploying Globus Compute Multi-User can address both the risks posed by the current science gateway access mechanism while providing all users with a new access modality. The ability to combine Globus Compute steps with reliable data transfer via Globus Flows has clear potential for science gateways, research instruments, and campus core facilities. And yet, adding a new service to an existing cluster requires planning, effort, and either new or repurposed resources to host the service. I’ll present how we’ve addressed these challenges to support early users of Globus Compute on Expanse.

The Globus development team will be available to answer all your questions about the Globus service. Table topics include data transfer and sharing, Globus Connect and Premium Connectors, the web app, CLI and SDKs, Automate, Compute and Authentication

Globus provides many capabilities that can reduce the barriers associated with AI research, development, and deployment, and in particular remove the frictions associated with exploiting diverse and distributed cyberinfrastructure. In this talk, we will describe how Globus is being used to support AI research generally and in domain-specific use cases. We will also present recent research that aims to further support AI use cases.

Diamond is a service designed to facilitate large model training across GPU clusters for scientists. It exposes web user interface to build container images, manage training jobs, monitor job progress, and manage data and provenance across clusters. Diamond relies on Globus Auth, Globus Transfer/Search, and Globus Compute for authentication, data management, and container/job management. So far, Diamond has been tested on NCSA Delta, TACC Frontera, and Lonestar6. We aim to release the alpha version of Diamond in early April and to support all NAIRR Pilot GPU resources.

Deploying secure privacy-preserving federated learning (PPFL) across distributed sites with large-scale, high-performance computational resources demands technical capabilities beyond the reach of many organizations. To democratize access to PPFL, we developed Advanced Privacy-Preserving Federated Learning as a Service (APPFLx). APPFLx simplifies cross-silo PPFL through an intuitive web interface for managing, deploying, analyzing, and visualizing FL processes. We leverage various Globus services to enable these capabilities. End-to-end secure federations are enabled using Globus Auth and secure and distributed training across heterogeneous compute resources is enabled by Globus Compute. With APPFLx, participants can establish or join federations using institutional credentials, perform privacy-preserving training on local datasets, and securely share model updates for aggregation. We present these capabilities in the context of enabling AI4Science use cases.

Additionally, we present the AI Data Readiness Inspector (AIDRIN) framework, designed to assess dataset readiness for AI applications, incorporating both traditional data quality metrics and AI-specific considerations. We discuss how integrating AIDRIN with APPFLx enhances data preparation and facilitates improved FL experiments, fostering trustworthy and effective AI model development.

This work addresses a critical need in the materials science community, which is the availability of persistent, easily accessible and useable machine learning models of materials properties that provide the user with predictions, uncertainties on those predictions (i.e., error bars), and guidance on domain of applicability to inform the user whether the model is reliable. We develop random forest models for 33 materials properties spanning an array of data sources (computational and experimental) and property types (electrical, mechanical, thermodynamic, etc.). All models have calibrated ensemble error bars and domain of applicability guidance enabled by kernel-density-estimate-based feature distance measures. All models are hosted on the Garden-AI infrastructure, providing an easy-to-use, persistent interface for model dissemination callable with only a few lines of python code. We demonstrate the power of our approach by using our models in a fully ML-based materials discovery exercise to search for stable, highly active perovskite catalyst materials.

The Rosen Center for Advanced Computing (RCAC) at Purdue University enables and facilitates research computing and data (RCD) workflows both on campus and nationally with our Anvil supercomputer as part of the NSF-funded ACCESS program.

Following the general availability of Globus Compute Engine with multi-user support we deployed GCE on Anvil (and soon all campus clusters) allowing for researchers to target Anvil as a compute endpoint and Globus Flows action provider using their ACCESS credentials.

Several opportunities are in the pipeline to build robust, scalable solutions on top of Globus Flows for both research projects as well as core facilities here at Purdue.

For the past few years our staff have collaborated with Danny Milisavljevic (notable Purdue Astronomer) on his global recommendation system (refitt.org) forecasting supernovae events using a distributed pipeline. Very soon, LSST will begin publishing events for which we will subscribe and stream data into our landing zone which we can build an automated system to trigger our inferencing steps.

Additionally, there is a project to rebuild the data infrastructure surrounding one of Purdue’s core facilities that operates some 40 instruments for life sciences. This represents a prototypical use-case for Globus Flows, including data transfers, transformations, archiving, sharing, and sign offs.

The A-KBS (Arctic Knowledge Based System) and Defense Resiliency Platform Against Extreme Cold Weather (DRP), are two DoD funded Arctic-focused projects in collaboration with CRREL, the Cold Regions Research and Engineering Laboratory (USACE). Over the past three years, our combined academic research teams across 5 Universities have been gathering data in Alaska using Globus to share data seamlessly via our portal. Now, we are expanding our work using Globus Compute towards embedding HPC into our Science Gateway leveraging our new Kubernetes cluster, AI/ML workflows, and supercomputing center collaborations. This presentation will share engaging media from our Alaskan fieldwork, discuss our predictive analytics and decision support tools, and provide an overview of how Globus serves as a key component of our high-performance computational cyberinfrastructure.

As AI-driven research grows, so do the challenges of managing, accessing, and sharing data across institutions. The National AI Research Resource (NAIRR) [1] aims to provide equitable access to data, computing, and software resources, but how well do current solutions align with researchers' needs? To answer this question, SGX3, the NSF Center of Excellence for Science Gateways [2], led a comprehensive effort to gather insights into the requirements for a NAIRR Portal. The effort included a large-scale survey with over 1100 participants, six online focus groups each two hours and a two-day in-person workshop. This presentation will share key findings from this engagement, revealing major challenges researchers face in data management, including barriers to FAIR (Findable, Accessible, Interoperable, Reusable) data implementation, difficulties with complex data transfer workflows, and the ongoing challenge to balance security with usability. Institutional silos complicate collaboration, making it clear that a more seamless, interoperable approach is needed.

This community-driven effort informs the needs how we can support better data management strategies, including the role of platforms like Globus in simplifying secure, high-performance data transfer. Additionally, we will highlight gaps that still need to be addressed, along with potential solutions in policy, infrastructure, and community engagement.

Attendees will gain insights into the evolving needs of the AI research community and how a well-designed NAIRR Portal can serve as a critical tool for advancing open, accessible, and efficient research data management.

[1] The NAIRR Pilot Portal nairrpilot.org
[2] Sandra Gesing, Claire Stirm, Michael Zentner, Maytal Dahan and Linda Hayden, “SGX3: Novel Concepts to Enhance Knowledge and Extend the Community Around Science Gateways”, Gateways 2023, Pittsburgh, PA, US, October 2023

The Research Data Archive (RDA; rda.ucar.edu) at the NSF National Center for Atmospheric Research (NSF NCAR) contains a large collection of meteorological and oceanographic observations integrated with NSF NCAR High Performance Compute resources to support atmospheric and geosciences research. Containing more than 700 dataset collections, the RDA supports the varying needs of a large and continually growing user community.

The RDA launched its integration of Globus data management services into its data portal in late November 2014, beginning with support of Globus transfer. In the ten years since, RDA users have used Globus to transfer more than 64 million files from the RDA, totalling greater than 15 petabytes of data. The RDA also leverages Globus Auth to enable user login with ORCIDs, and development is underway to use Globus Search to improve data search and discovery on the RDA web portal.

This talk will 1) provide a brief retrospective on how Globus has enhanced the RDA user experience by simplifying data management strategies and enabling scalable workflows, and 2) give an overview of current plans to transition the RDA into an open geoscience research data commons. This vision is guided by a recent workshop held at NSF NCAR to develop community requirements to modernize community-accessible data science infrastructure, to better connect geoscience datasets with geoscience-focused analytics environments, and to support researcher needs in meeting data sharing expectations in alignment with the FAIR (doi.org/10.1038/sdata.2016.18) principles.

The OpenCosmo project aims to provide an easy-to-use interface to extreme-scale cosmological data for scientists throughout the astrophysics community. Our platform relies heavily on Globus to manage the entire lifecycle of user requests, from initial authentication to delivery of the final results. Globus provides excellent tools for building user-facing applications on top of traditional HPC resources. In this talk, I review the architecture of the platform and how it could be used in other science domains. I discuss some of the challenges we have faced (and continue to face) and how the continued growth of Globus's abilities has the potential to revolutionize the way science is done in the 21st century.

The Globus development team will be available to answer all your questions about the Globus service. Table topics include data transfer and sharing, Globus Connect and Premium Connectors, the web app, CLI and SDKs, Automate, Compute and Authentication