Spark and Snappy Compression

Even on the latest EMR 5.13, spark cannot read Snappy compressed json files created by AWS Kinesis Firehose.

Example

>>> myfile = sc.textFile("s3a://mybucket/myfile.snappy") >>> myfile.first() Traceback (most recent call last): File "/tmp/zeppelin_pyspark-8372821998278831403.py", line 367, in <module> raise Exception(traceback.format_exc()) Exception: Traceback (most recent call last): File "/tmp/zeppelin_pyspark-8372821998278831403.py", line 360, in <module> exec(code, _zcUserQueryNameSpace) File "<stdin>", line 2, in <module> File "/usr/lib/spark/python/pyspark/rdd.py", line 1379, in first raise ValueError("RDD is empty") ValueError: RDD is empty >>> myfile = sc.textFile("s3a://mybucket/myfile") >>> myfile.first() u'{"Hello": "World!"}

This might be related: https://forums.aws.amazon.com/thread.jspa?threadID=251621

i3wm: Change display layout on dock and undock

I usually dock and undock my laptop several times a day. And because I'm using i3wm, gnome does not help with the reconfiguration of the displays when switching from the laptop display to the dock displays (I have two connected to the dock).

The simplest way I found to to solve this was to use autorandr mixed with srandr. This thread, on the old autorandr repo might give a few pointers.

The setup is simple:

Install autorandr (the python clone) from https://github.com/phillipberndt/autorandr or using

sudo pip install "git+http://github.com/phillipberndt/autorandr#egg=autorandr"`

Build and install srandr from https://portix.bitbucket.io/srandrd/? (you probably need to install libX11-devel and libXrandr so that you can build srandr)

Store configs running

autorandr --save mobile # dock the laptop and config as wanted with arandr autorandr --save docked

Start srandr with srandr autorandr -c by adding it to i3wm config file

echo -e "\n# Auto change display conf\nexec --no-startup-id srandrd autorandr -c\n" >> .config/i3/config

Lambda Architecture

Today I've learned about Lambda Archetecture while listening to an interview of Nathan Marz for Software Engineering Radio.

It's remarkable that a simple concepts like this one can bring so many benefits and solve so many problems when architecting a system that has to process "big data" in realtime.

Thank you Nathan for yet another tool on my engineering belt.

Shortening the gap between me and PhoneGap

I've just embraced on a side project: building an app for Android and iOS.

I must say I'm a bit scared because I don't realy have that much of an experience in mobile development.

Tring to find a framework to build the app on was not easy, but I ended up chosing PhoneGap from Adobe. It's just Apache Cordova with some suggar on top.

Lets see how it goes.

rmmod -f gallbladder

And here I am again. Feeling I have to write something, because, after all, that's the whole purpose of a blog.

The major thing that happened in my life since my last post was getting surgery to remove my gallbladder. It was relatively straightforward, and even though I was scared as hell, it all went ok. One of the side effects is that I cannot eat greasy food, or fat, or stuff like that, which, to a guy like me, is for the best.

On reading books

I think two things allowed me to read two books in the last 3 months (I normaly read about two a year): one was that I removed the TV from my bedroom, the other was that I got a Kindle Papperwhite.

Before getting the tv out of the bedroom, I would usually fall asleep watching tv. But now, without tv, I just read until I start skipping lines. The light in the kindle is great, because I don't have to leave the ceiling light on.

The last two books I've read are:

Outliers, by Malcolm Gladwell

The Myths of Innovation by Scott Berkun

They are both great, and I recommend them both.

Apache Storm

Just read this great post by Apache Storm creator, Nathan Marz on the history of Storm. So much hard work and attention was put in this piece of software.

I used Storm in my master's thesis. Storm allowed me to process, in real time, sales data coming from every one of Sonae's stores. Every item sold was ingested and analyzed by Storm and, using a Cassandra database, counters where updated that allowed managers to check, in real time, key performance indicators pertaining to sales and product figures.

Storm was robust, easy to use and configure, and I think I spend more time tinkering with the other systems I was using, like Camel, Cassandra and Redis, then actually implementing the topologies in Storm.

In the end, I had a nice, integrated solution, that fed some very cool dashboards that I built. I owe a significant part of my nice grade (and of the subsequent job ofer) to Storm and Nathan Marz hard work.

Organizing my work

I'm having a real hard time organizing and prioritizing the stuff I have to do at work.

I've tried various methods and I didn't really stick with any one of them, probably because of my laziness. Nevertheless, that laziness is what I want to fight, so I have to try and figure out something that works! This post is just so I can organize my ideas to try to figure out why the tools and methods I try are not working.

Here are some of the things I've tried:

Trello It's great: beautiful, powerful and at the same time simple. With a mobile app. But I tend to stop adding stuff to it and looking at it. Maybe because it doesn't integrate with Outlook, and my day is mainly decided by what mails I get. Having to transpose those requests I get in the mail to Trello is a PITA. What I like about Trello is that tasks don't have a fixed time slot, and that's nice, because everyday there's something urgent that comes up and my neat schedule goes bonkers.

Outlook Calendar Outlook is enforced by the company, so there is no way to get rid of it. But besides the meetings I have to attend, I tried to add the other stuff I have to do as an outlook calender entry. It doesn't work. Most of the time, I'm busy with something else at the time I set for the the entry, and I have to reorder my calender when that time comes up so I don't forget it...

Outlook Tasks I use the tasks in Outlook like a mini Trello. But it lacks most of the stuff that makes Trello great: There are no lists, it's tiny and ugly, you can inadvertently click a flag and the task just disappears. One thing that I like, though, is the fact that I can assign a task to an email. That's a really great feature.

Pomodoro Timer This is just so I can manage my time and stop iprocrastinating. The app is very nice but, for me, It doesn't always work because sometimes I ignore the breaks or I forget to start a new pomodoro.

Evernote One of the things I've managed to stick to is to write everything I do in a text file in Evernote. That is very useful when I have some doubt on something I did, but it doesn't help me organize what I have to do.

The company provides a project management tool (Artemis 7) to log some of the stuff I do everyday, particularly the things that are accountable and related to projects. The thing is, Artemis is only usable for a subset of my tasks, and because of that, I can't rely on it to manage my work. Also, it is very bad, as in "only works in IE" bad.

I would really, really love that the company was using something like Asana so I could manage all my work in one place.

New Nintendo 3DS

The new handheld console from Nintendo is still called Nintendo 3DS. This would be OK if Nintendo didn’t decided to have games that will work only in the new version of the console. How the hell will they differentiate those games from the older ones if both the old and the new consoles share the same name?

Fafião's natural pools | Piscinas naturais de Fafião

We went to these marvelous natural pools in the middle of Gerês. The water is crystal clear and the rocks smooth and milky. You can even use some of them like a water slide! The place is easy to reach, just outside the village of Fafião. I had a hard time getting the GPS coordinates online, so I'm writing them here, let's just hope google finds this.

Piscinas naturais de Fafião | Google Maps

GPS Coordinates: 41.7093406, -8.0936247

Photos by Milai

Cheaping out

I've just closed my dreamhost hosting account. They where great and never failed me but I just couldn't justify spending $10 a month on something I used so little. I don't have any professional website, I just had some old wikis (one for my projects and one for my dissertation), a blog and a landing page with my name and contacts. I ended the wikis, moved my landing page to github.io and replaced the blog with this tumblr. I'm still using dreamhost as my dns/registrar so I can have my custom domains pointing to the blog and to my landing page.

Summing it up, my internet presence is dominated by two domains:

ctrler.eu for this tumblr.

rteixeira.eu for my personal landing page, where you can get links to flickr, facebook, github and other stuff.

If I need new pages, they will hopefully be static ones, and for that I'll just use github pages. If I need dynamic stuff, I'll go for something like NearlyFreeSpeech

Connecting Weblogic with Microsoft SQL Server 64bit via XA Datasource

I had a bit of trouble getting a XA JDBC to connect to a 64bit Microsoft SQL Server. The instructions on the Oracle KB are not clear that one has to rename the dll file to sqljdbc.dll uppon placing it on the SQL Server's binn folder.

The appropriate dll is shipped with Weblogic and can be found in the following folder (for Weblogic 11g): $ORACLE_HOME/oracle_common/modules/datadirect_4.1/

There is one file for each architecture:

sqljdbc.dll: for 32-bit Windows

64sqljdbc.dll: for 64-bit Windows (Itanium)

X64sqljdbc.dll: for the X64 processors

Again, one of these has to be renamed to sqljdbc.dll and put inside the SQL Servers binn directory.

In Weblogic, to create the datasource, one should choose:

Datasources > New > Generic Data Source > Database Type: MS SQL Server > Database Driver: "*Oracle's MS SQL Server Driver (Type 4 XA) Versions:7.0 and later"

A very original project from two New Yorkers, Bekka and Dave, who go out every wednesday to interview a random person on the streets, taking their picture and asking them what are they doing.

Parker Jotter

I've bought a Parker Jotter today.

It was the pen that was used by James Bond in the movie GoldenEye. It's history is much richer than that, though. I remember being a kid and "borrowing" a few of my fathers Parkers. They always felt special and refined.

The fine refill writes pretty well on my moleskine and, because it is the all metal version, there is no chance I'll chew it when I get nervous.

A beautifully made microsite that is a great resource to understand Kowloon Walled City: its history, its origins, its inhabitants and its day to day life. Wall Street Journal compilation of images, videos and other information about the city is very engaging. Having read about the Kowloon Walled City before, it has always amazed me how such a thing could have existed through most of the 20th century, only being brought down in the 90's.

This LyX thing looks pretty nice. I wish I knew about it when I wrote my thesis.

Installing more than one Oracle product at the same time using XMing

I usually have to setup more than one managed server at the same time when preparing a Weblogic Domain or a SOA Suite installation.

To install Oracle software I normaly use XMing so I can have the installation GUI on Windows. Because of a bug in some Oracle GUIs, I have to use XMing without the multiwindow option. Without that option I get a single Xorg root window with all the windows inside, but because I have no window manager, all the windows appear on top of each other and I can't even drag them to rearrange them! That makes the "windows" part of XWindows a bit useless. :)

The solution is simple, just start a simple window manager like TWM in the background before starting the installs. It doesn't matter in which of the hosts you run it, it will manage all the windows for all the installs across all the hosts, as long as the GUIs share the same XMing root window.

Example:

twm & java -jar wls1036_generic.jar