Databricks accelerates time to value at scale with ngrok

Secure connectivity is a must-have requirement for Databricks customers

Enterprises use Databricks to process, analyze, and monetize their datasets with solutions from business intelligence (BI) to machine learning. The architecture entails Databricks backend services connecting to the infrastructure deployed into the customer’s cloud accounts to process and store data. Thus delivering value to customers required connecting into customers’ networks. 

"For enterprise customers, security is non-negotiable," says Ihor Leshko, Director of Engineering at Databricks. "While our architecture was secure, we needed additional capabilities to support Databricks deployments in the largest organizations and eliminate the need for public IP infrastructure."

With security being table stakes, Databricks needed to remove this particular point of contention with enterprise prospects and find a solution to remove public IPs from worker nodes.

Searching for the optimal solution

Databricks evaluated seventeen alternative solutions - none of them provided robust security in a scalable manner in the way that ngrok did.

"We did a very broad search for solutions from major cloud providers and third party vendors," says Leshko. “The services we saw on the market either had serious reliability challenges or did not allow us to configure security in a granular way that met the needs of our customers. We thought of building a relay service ourselves, but we still couldn't help but wonder if someone else had solved this before."

After spending months researching alternative solutions, the team nearly resorted to building a solution in-house – until they discovered ngrok on GitHub.

“We came across ngrok on GitHub. We looked at how it was being used and thought ‘This seems like exactly what we need.’ All of the features were out of the box,” says Leshko. “It supported TCP connections, HTTPS connections, websockets – it seemed like it could do everything and more.”

Discovering the power of ngrok

A few weeks later, Databricks experimented with ngrok’s managed offering at a Hackathon.

“The team pulled off a crude demo with ngrok and saw how simple it was to integrate with,” says Leshko. “We were able to eliminate public IPs. Within two to three days, people were getting excited about it.”

When evaluating whether to build or buy, Databricks prioritized delivering immediate value to their customers. 

"While there was a lot of work to do, we had two choices: we can build it on our own, or we can take advantage of this technology and deliver value right now,” says Leshko. “We could spend all of this time tailoring for exactly what we need and develop it natively on our infrastructure, or take something off the shelf and not wait to onboard our enterprise customers."

Within just three months Databricks was able to start previewing the product with their enterprise customers.

“Alan and the team were great,” says Leshko. “They were super responsive to our asks, ensuring we had what we needed to be successful.”

Powering deployments with ngrok

Since 2019, Databricks has used ngrok to solve the challenge of ingress into their customer environments.

“ngrok opened the door for Databricks to quickly onboard these big customers,” says Leshko. “With ngrok, we get comprehensive security with a full scope of functionality for all of our use cases, and for larger deployments, it gives us a scalability advantage, allowing us to eliminate much of the operational complexity.”

ngrok connects with customer environments through a single point of ingress to enable all functionality, and in turn, massively simplifies connectivity. And because ngrok provides a secure pathway, security reviews and deployments with customers are infinitely faster.

Today, Databricks meets the security and compliance requirements in highly regulated industries, such as financial services, healthcare, and telecommunications.