Integrating encrypted data analysis with Databricks

“Roseman Labs' integration reduces manual processing efforts by more than 90%, cutting the time from 48 hours to just 90 minutes.”

We had the pleasure of interviewing a user who has integrated our software with Databricks, enhancing their workflow and data analysis while ensuring sensitive data remains secure.  

This article explores that workflow and the benefits of connecting multiple software systems. Learn how this integration saves valuable time and effort while generating the greatest insights. 

The Roseman Labs setup in Databricks

1. Configuration in Databricks: Integration begins with setting up a dedicated workspace in Databricks. This workspace is tailored to store all configurations (including Roseman Labs’ connection files and cryptographic keys) ensuring secure data management and streamlined access.

2. Processing and automation: Following configuration, the Roseman Labs engine takes over data processing tasks, performing data merging and calculations. Aggregated results are saved as CSV files in Databricks and then transferred to designated secure storage.
   
   Routine data processes are automated using Python scripts within Databricks. These scripts, coupled with a rosemanlabs_utils module, handle various automated tasks and maintain environmental consistency through controlled updates. 

3. Data visualization: To effectively visualize data outcomes or data trends over time, DeltaViews are generated. Connected to a PowerBI dashboard, these visualizations offer dynamic insights from real-time analysis, providing business stakeholders with a clear and timely overview of data progression. 

Automation and efficiency

Automation is key in this workflow, significantly reducing manual effort. Each month, the user processes three data requests using the Roseman Labs platform. By using the data request, more than 20 organizations are providing their data in a structured and encrypted manner. The next step is updating Python scripts in Databricks, each dedicated to specific tasks, and ensuring that the correct reference to the tables is in place. 

The system uses Git for precise version control and script management, facilitating automated updates and consistent operations across environments. The rosemanlabs_utils module plays a crucial role in maintaining path and environment variables, ensuring seamless operation following updates. 

Technical advantages

This workflow showcases how effectively multiple software systems can be integrated to streamline operations. For the user, integration with Roseman Labs significantly reduces manual processing time from two days to roughly ninety minutes.  

In conjunction with Databricks’ data management capabilities, Roseman Labs’ encrypted computing allows for secure integration of sensitive and public data sets, and results in a data analysis process that is both fast and secure.

Automation saves time, increases the system's reliability and maintains data integrity across updates, contributing to a secure, efficient analytical environment. This approach is an ideal model for organizations that manage sensitive data who want to extract valuable insights while maintaining high security standards. 

Curious how Roseman Labs can be integrated in your workflows? Send us a message!

Generate new insights on sensitive data with Roseman Labs’ secure Multi-Party Computation technology. Want to find out how your organization can do that? Contact us using the form below.