Next-generation Data Pipelines

| Comments

It’s Q4 of 2018 and it’s really interesting to observe the change in Big Data Landscape, especially around open-source frameworks and tools. Yes, it’s still very fragmented, but the actual solutions and architectures start to slowly converge.

Right now I’m in the beginning of a huge platform redesign at work. We always talk about various frameworks and libraries (which is actually just an implementation detail), but I started to think: what qualities should modern data pipelines have going forward? The list that I came up with is below.

API Flexibility ∝ 1 / Data Definition Strictness

| Comments

An interesting observation based on the recent conversations at work: the more strict the data format used in the API definition, the harder it is to change the API behaviour later. And vice versa, it’s easier to change the APIs that use flexible data formats.

The Level Below

| Comments

Over the years I realized a very simple, but also a fundamental principle about being a better software engineer – understanding what’s happening one level below. With level I mean any level of abstraction you operate, for example HTTP API for a Front-end engineer, JVM and its internals for an enterprise Java developer, etc.

It sounds like an obvious suggestion, but it’s actually very useful to apply this principle to any new feature you’re going to work on.

QCon London 2018

| Comments

Last week I had a chance to attend and speak at my first QCon conference: QCon London 2018. QCon has been an example of an extraordinary tech conference for me – great organization, amazing lineup, great tracks covering bleeding-edge tech, as well as various best practices around distributed systems. Also, they managed to get a decent vegetarian food right 😉

Keep reading if you’re curious about the best talks I attended and my speaking experience.

Message Enrichment With Kafka Streams

| Comments

I’ve been working with Kafka Streams for a few months and I love it! Here’s the great intro if you’re not familiar with the framework. In the sections below I assume that you understand the basic concepts like KStream, KTable, joins and windowing.

Message enrichment is a standard stream processing task and I want to show different options Kafka Streams provides to implement it properly.

2017 in Review

| Comments

Another “Year in Review” post, again mostly focused on professional life & tech stuff (since it’s a tech blog 😉). Check 2016 here.

A Letter to You, %software_conference_organizer%

| Comments

I’ve been a frequent software conference attendee and a speaker for the last few years. I’ve seen great events with amazing speakers, facilities and food. And I’ve seen very poor events with a lot of issues like zero diversity and really weak speakers.

And I’ve been recently thinking about some recommendations for the future events and came up with a list below. I emphasize the basic diversity and inclusion rules, because our industry constantly lacks them.

6 Months at Demonware

| Comments

You can probably decide I’m busy based on the absence of any new posts since July :)

And indeed, I was pretty busy for a while. I started to work at Demonware precisely 6 months ago and it’s being an amazing ride so far.

Bash Scripting Best Practices

| Comments

I was always afraid of writing shell scripts. Bash seemed to be a programming language that doesn’t allow a slightest mistake… Extra space here and there and everything blows up.

Like with every skill, persistence and repetition help. I’ve started writing more and more bash scripts a few years ago. But it’s important to remember one simple rule – when things become really complex you need to switch to Python/Ruby/scripting language of your choice. Please do!

Anyway, today I want to share some of the very practical conventions, best practices and recommendations I gathered over these years.

It’s not an introduction to bash, you should have some background already (ideally some war stories as well).

Also, I’m not an expert! It’s ok to not agree with me. And I’m pretty sure almost everything I mention can be improved. So please help me and leave some feedback ;–)

How to Build CLI in Node.js

| Comments

CLI (Command-line interface) is a very common way to run a program in a terminal. As a software engineer you use different CLIs every day – git, Docker, npm, etc.

Today I want to share my experience building CLIs using Node.js and a few helpful packages.