Load testing fluentd with wrk2 and OpenResty

We’ve written some complicated transform rules for fluentd to add fields to the record, serialize it with Apache Avro and dump it all into kafka. That was fine, until some service owners said “Hey, we don’t want to learn this Avro stuff, just let is write to fluentd on UDP” Management decided we should load test fluentd, so the ticket I got was:

 

“The test ideally should be UDP events sent to localhost at a rate of 1,500, 3,000 and 10,000 events per second.”

 

Great, so how do I manage that? A colleague told me about wrk2, which generates HTTP requests at precise rates, is fast, and has a low system impact. The thing is, it ONLY generates HTTP GET requests, and it’s written in C, which I wasn’t about to bother learning for this one-off effort.

 

Enter OpenResty. If you’re not familiar, OpenResty is nginx with a LUA interpreter built in, so you can write LUA scripts to handle requests. There are a lot of things OpenResty is wrong for, but this was one of the things I thought it could do well. Fast, multi-threaded, efficient. What does that have to do with “UDP events sent to localhost”? Well, I wrote just enough LUA to do  that, based on GET requests generated by wrk2. The LUA script is here if you want to have a look. It returns 503 if it can’t send the request. In the code, you’ll see reference to “somefile.json”, that’s the mock payload which is sent to fluentd.

 

I start wrk, generating 1500 requests a second, with 10 threads, 10 connections to nginx, for 10 minutes:

./wrk -t10 -c10 -d600s -R1500 http://localhost:10082/udp

 

The results were interesting. fluentd is single threaded, so it just ran the core it was on at 100% while it read UDP packets out of the kernel buffer, but, it didn’t drop not one. When I ran it for 60 seconds, it generated 90,000 events and sure enough I got 90,000 Kafka messages out the other end of the pipeline. Way to go Ruby!

 

Advertisements

About robertlabrie
DevOps Engineer at The Network Inc in metro Atlanta. Too many interests to list here, check out my posts, or look me up on LinkedIn

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

%d bloggers like this: