Why You Can’t Do All of Your Data Engineering with SQL
There is a common misunderstanding in data engineering that you can do everything you need to create a Big Data data pipeline with SQL. This notion is being promoted by some vendors and companies. They’re wrong and you can’t do all of your data engineering with SQL. You will eventually need a programming language to […]
Thoughts on Cloudera Merging/Buying Hortonworks
Cloudera has merged with/purchased Hortonworks. As a former Clouderan, it’s interesting to see this move on several levels. I’m going to share my insights from the outside as a former insider. Full Disclosure: Although I’m former Cloudera, I don’t own any shares of Cloudera or Hortonworks and don’t plan to purchase any in the short-term. […]
Creating Work Queues with Apache Kafka and Apache Pulsar
A common use case for using Kafka and Pulsar is to create work queues. The two technologies offer different implementations for accomplishing this use case. I’ll discuss the ways of implementing work queues in Kafka and Pulsar as well as the relative strengths of doing each one. What are work queues? A work queue is […]
InfiniteConf Keynote – Why Real-time is the Future
Here is my keynote from InfiniteConf 2018. I talk about why real-time is gaining so much momentum, what it does for businesses, how it helps data sciences, and some common use cases.
What is a Data Pipeline?
I’ve been seeing some questions about data pipelines lately. I realized I haven’t written a post that gives the level of detail necessary for a good definition of a data pipeline in the…
Professional Data Engineering Review – Sanjoy Roy
Note: this is a guest post from Sanjoy Roy who is reviewing my Professional Data Engineering course. Since late 2014, I have been drawn into various analytics projects which required a good mix of skills for both data engineering and data science. There are a lot of good MOOC available now which covers very focussed […]
Saying You Have Small Data Isn’t Belittling Your Use Case
There is a common beginner question for engineers starting out with Big Data. An engineer will do a post to a social media site saying “I need to know which Big Data technology to use. I have 3 billion rows in 10,000 files. The whole dataset is 100 GB. Is Big Data Technology X efficient […]
The Two Types of Data Engineering
There are two different types of data engineering. There are two different types of job types with the title data engineer. This is especially confusing to organizations and individuals who are starting out learning about data engineering. This confusion leads to the failure of many teams’ Big Data projects. Types of Data Engineering The first […]
Why Real-time is the Future
One of the benefits of teaching and consulting is the sheer number of organizations, teams, and people I get to work with. Since I deal with so many different groups, I can see patterns emerge much faster than others. One pattern I saw early on was real-time Big Data. Organizations wanted to do things in […]
The Four Types of Technologies You Need for Real-time Big Data Systems
Creating real-time data pipelines bring new challenges. There are new concepts and technologies that you’ll need to learn and understand. To help you understand the basic technologies you need in a real-time data pipeline, I break it down into 4 general types. These types are: Processors Analytics Ingestion and dissemination Storage Processors A processor is […]