Faça uma busca
Faça uma busca

IDAC

As part of the agreement with the Stanford Linear Accelerator Laboratory (SLAC), LIneA began the deployment of a data center in 2021, known within the LSST project ecosystem as the Independent Data Access Center (IDAC). The IDAC-Brazil will be part of a global network composed of a dozen centers across all continents, hosting a portion of the data, distributing access to analysis tools, and providing additional computing resources to the collaboration.

Having an IDAC in Brazil is strategically important as it facilitates access to data and computational infrastructure for Brazilian researchers. Additionally, it encourages the local development of scientific platforms and analysis pipelines, promoting technological exchange with other centers in the network.

The Brazilian IDAC will be of the “Lite” type and will host catalogs generated from “co-added” images, the number of columns and the volume of which are yet to be defined. In addition to this catalog, hosting other necessary analysis products is also planned, such as depth maps, footprints, and photometric redshift tables, which will be shared with the collaboration. These tables are part of LIneA’s in-kind contribution to LSST. Although it is not a requirement of the Lite IDAC mode, there is a possibility that the Brazilian IDAC will also store the co-added FITS images to serve applications that require these images or to offer local access to the images through the visualization tools of the LIneA Science Platform.

The IDAC will maintain at least three versions of the catalog. Two of them will have restricted access for LSST members (the one released in the current year and the one from the previous year), and the third, which was released more than two years ago, will be made available to the general public in accordance with the internal rules of the Rubin Observatory. 

The data will be available in two ways:

  1. In the database for exploration through the LIneA Science Platform. This platform will provide a JupyterHub environment for data analysis, tools for image and catalog visualization, “cutout” services (showing cropped images of target objects), and “cross-match” services (cross-referencing between two catalogs covering the same region of the sky). 
  2. In files saved on disk, either in the original format or in the “hipscat” format — a format developed by the LSST Interdisciplinary Network for Collaboration and Computing (LINCC) aimed at optimizing the cross-match service.

In addition to the portfolio of services offered by an IDAC Lite as defined by the LSST project, LIneA will curate additional data from legacy projects obtained at different wavelengths. Among these, the catalogs from the Dark Energy Survey and the Sloan Digital Sky Survey, which are already maintained by LIneA, will be available, as well as others of interest to the community (e.g., GAIA, VISTA, 2MASS), to be determined through surveys conducted within the community.

The IDAC infrastructure will be part of the multi-user e-Science center operated by LIneA, which will prioritize LSST project members on certain equipment acquired as part of the agreement with the Rubin Observatory. 

The specific IDAC infrastructure will consist of the following equipment:

  • Database – two PostgreSQL servers with 500 GB of memory and 500 TB of storage, and one dedicated ingestion server.
  • Storage System – Approximately 1 PB .
  • Lustre storage.
  • NAS Storage System – Approximately 4 PB of storage.
  • Cluster – 500 cores from the HPE Apollo 2000 cluster.
  • Kubernetes Cluster hosting Jupyter Notebooks.

For members of the Brazilian Participation Group (BPG), LIneA will provide 500 processing cores from the HPC Apollo cluster and coordinate proposal submissions, offering technical assistance for using Santos Dumont. The HPC service may also be made available to the broader community upon specific requests.

To ensure the quality of the service, LIneA has commissioned RNP (Brazil’s National Research and Education Network) to develop an improvement plan for its external and internal network, which includes provisioning 100 Gbps connections. To achieve this, a new router is being acquired, and the existing DMZ equipment is being upgraded for international transfers, along with new switches and transceivers for the internal network.

In addition to the LIneA Science Platform, the IDAC will maintain other platforms, including:

  • PZ Server
  • LIneA Occultation Prediction Database
  • Cluster Analysis and Visualization Service (CAnVAS)

All of these are currently in development by LIneA’s IT team with support from the INCT of e-Universe.

The start of IDAC’s operation is scheduled for 2026 when the first survey data becomes available. Therefore, the next two years will be used for the gradual deployment of the center, commissioning using simulation data, tests conducted on the telescope, and training of the Brazilian community.

The deployment of IDAC is being supported by FINEP, CNPq, and Faperj through the INCT of e-Universe.