Techstrong TV: Announcing Project Shapeshift Milestones

Imply will be announcing Project Shapeshift Milestone 1 which includes the launch of a new product and cloud service, as well as a private preview of a major architectural expansion of Apache Druid. This announcement builds upon Imply’s November 2021 announcement of Project Shapeshift, a 12-month initiative to tackle the most pressing issues that developers face when building modern analytics applications and to transform the Druid experience across three dimensions: cloud-native, simple and complete. The video and a transcript of the conversation are below.

Recorded Voice:     This is Digital Anarchist.

 

Alan Shimel:            Hey everyone, welcome to another Techstrong TV segment. I have a new company with a – it’s not a new open-source project, it’s actually a very popular open-source project, but one that we’ve never covered here on Techstrong TV. Let me introduce you to Vadim Ogievetsky, am I close Vadim?

 

Vadim Ogievetsky:  Close enough.

 

Shimel:            Okay. Well say it right anyway though, because someone from your family is listening and they want to hear it.

 

Ogievetsky:  Yeah. Well everybody. My name is Vadim Ogievetsky.

 

Shimel:            Ogievetsky.

 

Ogievetsky:  I am a cofounder here at Imply.

 

Shimel:            Yes. So, so Vadim we’re going to get into Imply in just a second, but you know it’s hard to tell the Imply story without talking a little bit about Apache Druid. And if you don’t mind I want you to start there.

 

Ogievetsky:  Absolutely.

 

Shimel:            You know you were very involved in founding this project, tell us about it.

 

Ogievetsky:  Absolutely. So before me and my cofounders founded Imply, we were working all in a company that was building an analytics product for the ad tech market. When I say an “analytics product” that’s really something that takes a lot of data and can answer questions like, “Who are my top publishers?” “What are the biggest campaigns that are trending right now?” “Which campaigns have suddenly maybe lost their steam and we need to like dive in and investigate those campaigns?”

 

                                 And we were building this application, something that you can provide to the users of, like to the publishers who give it out to the actual people that are using these apps. We really stumbled upon a problem that we couldn’t find a tool, a database that would be able to power, to be able to consume that much data and at the same time answer those questions in a way that can provide an interactive experience to people.

 

                                 And that was kind of like our ah-hah moment where we start – we basically said, “Wow there’s nothing that could solve that for us without bankrupting the company,” because all the things that were available at the time were from very expensive proprietary vendors.

 

                                 So we started building a new database, which you know looking back on it sounds a little bit crazy, but we really sit in the open source and it, it took off. It was really cool. We build our app on top of it and once we open sourced the project we also saw a lot of other people come in and start building other apps on top of it.

 

                                 As I said we were kind of building an analytics application for the advertising market, but we saw other companies pick it up and start using it for things we never even envisioned it for originally. Notably Netflix was, who picked it up and started using it for all of their user-facing analytics. So any interaction that you do on the Netflix site their product managers, their engineers when they want to understand how people are engaging with their site they’re looking at an application that’s powered by our database.

 

                                 We saw people starting to pick it up in supply chain monitoring, InfoSec, networking monitoring, really you name it. And we just saw that there is this broad market for applications that are interactive, that are delivered to the end user that need to support heightened currency. We decided to make a company around that.

 

Shimel:            Sure. So Vadim give me timeframe. When, when did – and by the way this database we’re talking about became Apache Druid.

 

Ogievetsky:  Exactly.

 

Shimel:            The Apache Druid. What, what, when was this that you, you and your partner or cofounder kind of said, “We’ve got to build something”?

 

Ogievetsky:  I mean we’re coming up on ten years of open sourcing Apache Druid, which has been quite a journey and it’s been very exciting.

 

Shimel:            Yep and that’s why I wanted you to bring it up, right, because there’s, look, there are a lot of open-source databases out there that are very scalable . And you know we, today, in today’s world we see graph databases and time series databases and of course there’s NoSQL and SQL and, and so many different things. But I want our audience to recognize that this was ten years ago. A lot of the, the stuff that you see people playing with now didn’t – wasn’t even in existence then. So and you know why did you build it from scratch? Because I mean had you had the foresight to say, “Okay, we’ll wait ten years and maybe something else or someone else will do.” You don’t have that luxury, right?

 

Ogievetsky:  And we were as I said we were working with the advertising industry then and I think in many ways with digital advertising they were kind of ahead of the curve on just the scale of data that they need to analyze.

 

Shimel:            Yeah, yeah.

 

Ogievetsky:  Now, hearing companies suggest that they a lot of data, that they want to offer analytics to their customers, that’s really that’s here and now, that’s the reality of the industry. But ten years ago that wasn’t the case and I think we happened to be lucky to be in a place of an industry where it was kind of like pushing ahead of the rest.

 

Shimel:            Fair. Just you know to give them – do your cofounder also is the person who helped you found what became Druid. His or her name is?

 

Ogievetsky:  My two cofounders are Fangjin Yang and Gian Merlino and we all work together on Druid from day one.

 

Shimel:            Got it, okay. So now let’s fast forward, you guys decide – okay you’ve got Druid, a lot of people pick-up and it becomes part of the Apache Foundation, which is kind of a you know a really big stamp of approval and it gives it a lot of legitimacy in the eyes of our audience and the eyes of the developer community. When did you guys decide to do a, a commercial company around it?

 

Ogievetsky:  Yeah, well when we, when we open-sourced Druid and we saw it being picked up by companies outside of the industry we were kind of originally developing it for, which was advertising analytics, when we saw people say, “Well you know this analytic used case, this pattern that you’re solving this applies to my industry as well.” This is, this is one of the awesome powers of open source, because I mean we were just a couple of engineers working in a company.

 

                                 We didn’t have like the budget or time to do like market research on the applicability of these techniques, but the market kind of did it’s research for us and kind of came to us by, by actually setting up and utilizing this, this technology. And before we know it we were going all around Silicon Valley kind of giving tech talks about Druid and helping people set it up in their, in their customers, in their data centers. We just we were very excited about the project.

                                 We wanted to build a company that could both nurture Druid and help it grow and develop into a more mature project and also offer true value add on top of it and we’ve been doing that ever since.

 

Shimel:            Excellent. Okay and then I just want to make sure we get it all for our audience so they – we don’t lose them Vadim. The name of the company that you and your cofounders started is Imply, yeah?

 

Ogievetsky:  Yes, Imply, yeah.

 

Shimel:            And the website?

 

Ogievetsky:  So we’re not Druid company, you know the Druid trademark is owned by Apache Foundation and we love them for that. You know we are Imply, Imply Data and we provide a whole suite of things to make building applications on top of the Druid as simple as possible.

 

Shimel:            Excellent. The website for Imply Data?

 

Ogievetsky:  Sorry, come again?

 

Shimel:            The website for Imply Data?

 

Ogievetsky:  Imply.io.

 

Shimel:            Imply.io, so that’s spell it: I-M-P-L –

 

Ogievetsky:  I-O. I-M –

 

Shimel:            So I know .IO, that we’re familiar with but how do you spell Imply?

 

Ogievetsky:  I-M-P-L-Y.I-O

 

Shimel:            Right, okay. People got it. I’m sorry Vadim, I – something’s we know it’s basic block and tackling here and so we make sure. Because I want to a make sure people are able to go check it out.

 

                                 All right, so now we, we have Imply.io, you’ve built, you make it easy for people to build applications on top of the Druid database, yes.

 

Ogievetsky:  Yeah and now we’re kind of taking it to the next level. So we’ve always kind of added value with UI’s monitoring tools, deployment infrastructure and now we’re announcing Imply Polaris, which is a SasS offering of Druid. It’s not just Druid in a cloud, it integrates so much more, including all the monitoring that you need and push-based streaming ingestion by our partnership with Confluent Cloud.

 

Shimel:            Got it. So it’s called “Polaris,” yeah?

 

Ogievetsky:  Yes.

 

Shimel:            Okay. And really it’s, it’s sort of, it’s, it’s Druid-Plus as a service, right?

 

Ogievetsky:  Absolutely.

 

Shimel:            So you get the basic. I mean obviously if you have a database, it’s everything that exists onto the database, the OS infrastructure, all that. But then on top of the database you have some other kind of I don’t know if they’re open source or not, but other extensions and capabilities that you don’t get just with like the, the vanilla Druid if you were setting it up yourself.

 

Ogievetsky:  Absolutely. On top of the database we have a suite or proprietary tooling that kind of comes bundled to you from the get-go when you use Polaris. That includes our data exploratory UI Pivot, it includes Clarity, which is a monitoring tool we have specifically designed for Apache Druid, and also all of the deployment and infrastructure is handled for you. I mean it’s serverless so you don’t have to think about servers, which is a really neat concept.

 

                                 You know if you were deploying Druid yourself as people have been doing up to this point, even if you’re using our tooling, you still have to think about like, “How many servers do I have running,” like provision use servers, you’re still talking about in that language and we’re kind of moving beyond that now, which is extremely exciting. It’s just an EPI.

 

Shimel:            So but from in my mind though the question then begs how do you price this? Because so many of these kinds of things were priced on CPU servers, you know space and such, how, how are you pricing Polaris?

 

Ogievetsky:  We have a pretty industry standard competitive pricing that is basically based on a set size that it incorporates and then a lot of variables, like the number of users, the number of, the amount of data, and it, it’s kind of like a package that you can kind of choose which one you want.

 

Shimel:            Got it, good. I just want to make sure for anyone who want to go check it out, if they go to imply.io it’s right on the front page there?

 

Ogievetsky:  Oh yes, absolutely, yeah. So imply.io will, will take you to wherever you need to go.

 

Shimel:            Okay. So Vadim now let’s talk so – what, give us the kind of use cases for Polaris. I assume company – you know you tested it, you beta it and you’ve got customers on it. What are some of the use cases people are using it for?

 

Ogievetsky:  Absolutely. I mean it’s, it’s really used when you have some developer – just like us back in the day when we were building out this application that we ended up building Druid as a result. If you have a developer that’s tasked with or you know an architect that’s tasked with creating an analytics application that you can deploy to you know thousands of people, that will consume terabytes, terabytes and maybe even petabytes of data and that needs to be very quick and interactive.

 

                                 The use case is that you can basically point your data stream at Polaris and build your application on top of our analytics _____. This is, this is the fundamental thing we want to empower. We want to make that journey that we have to like undergo and it took us years and years, something that now can be done in days or hours.

 

                                 In terms of the applicability of this like I literally see no end to where, where this applies from you know advertising, kind of where we started, to supply chain monitoring, network analytics, user behavior monitoring and tracking, fraud detection. Really I’m never, I never cease to amaze of how many cool different use cases people are using the software for.

 

Shimel:            Absolutely. And you know it’s interesting… I mean look my experience with open source is once this out here now there’ll be, you’ve got more use cases, because just like when you guys started Druid itself you were thinking in terms of you know advertising and you know clicks and so forth. But Netflix uses it for something else and someone else uses it for something else. And that’s just the beauty of, of that open source model is you know it’s like giving people clay and they mold it into what they, what they did and what they want.

 

Ogievetsky:  Absolutely. It’s one of the best aspects of this job to see how people put Druid to creative uses, kind of… And it’s most exciting especially when people reach out and you know a lot of people kind of don’t like to announce themselves until they have something that’s ready and good to go and sometimes we kind of stumble upon people… You know we have a very active community channels.

 

                                 We really engage with the community and we help them out, but sometimes people come to the community channels and they have like a ____ to complete, like application that they build and it’s very exciting. Oftentimes they, they have this built out and they’re just looking for help monitoring and making sure that they can meet their SLAs and making, taking the cost out of deploying it and everything like that and that’s where we step in.

 

Shimel:            Absolutely. You know Vadim this is great. I just want to make – it’s available now. People watching this today can go check it out and –

 

Ogievetsky:  Yep, yeah. People can, can –

 

Shimel:            Is there a free or a trial version?

 

Ogievetsky:  Yes, there is. So if you go to imply.io you can check out the trial version. And also to people that are just discovering Druid for the first time maybe you can download that straight off the Apache website and, and play with that as well.

 

Shimel:            Excellent. Vadim, thanks for coming on the show. We’re about out of time, but best of luck with Polaris and I, I think it will be interesting to see how the market you know takes Druid as a service or Druid Plus as a service if you will. That will be interesting to see what happens. Thanks for coming on the show.

 

Ogievetsky:  Thank you very much for having me on Alan.

 

Shimel:            All right. Ogievetsky, better this time?

 

Ogievetsky:  Yes, nailed it.

 

Shimel:            [Laughs] All right. Check out imply.io: IMPLY.IO for the Polaris, Druid, Druid Plus as a service.

 

                                 We’re going to take a break here on TechStrong TV. We’ll be right back.

 

[End of Audio]

 

 

Alan Shimel

As Editor-in-chief of DevOps.com and Container Journal, Alan Shimel is attuned to the world of technology. Alan has founded and helped several technology ventures, including StillSecure, where he guided the company in bringing innovative and effective networking and security solutions to the marketplace. Shimel is an often-cited personality in the security and technology community and is a sought-after speaker at industry and government conferences and events. In addition to his writing on DevOps.com and Network World, his commentary about the state of technology is followed closely by many industry insiders via his blog and podcast, "Ashimmy, After All These Years" (www.ashimmy.com). Alan has helped build several successful technology companies by combining a strong business background with a deep knowledge of technology. His legal background, long experience in the field, and New York street smarts combine to form a unique personality.

Alan Shimel has 54 posts and counting. See all posts by Alan Shimel