Fenway 2016

fenway park

Fenway Park from the grandstands (section 30)

Baba took the whole clan to Fenway to see the Red Sox play the Chicago White Sox.  It was an afternoon game so that the kids could come, too.

del's lemonadeBaba generously bought us stadium food as well – no small sum for hot dogs, pretzels, pizza, and Del’s Lemonade.

The Sox stunk until the bottom of the 5th inning, at which point they came back from 4-0 lead by the White Sox to win 7-8.

Delta fell asleep by the 8th inning, which is surprising given the amount of noise every time the Sox got a run.  Kappa, who’s about a year and a half, stayed awake and in mostly good spirits through the entire game.  Beta was well behaved, and Alpha genuinely enjoyed herself.

We left as the 10th inning was starting so that we could avoid some of the crowds with the kids and headed down the street to get dinner at Wahlburgers.

“ERROR: … failed to process… ” in NiFi

I was greeted by a few cryptic things in NiFi this morning during my morning check-in.

  1. A PutSQL processor was reporting an error:
    "ERROR: PutSQL[id=$UUID>]failed to process due to java.lang.IndexOutOfBoundsException: Index: 1, Size: 1; rolling back session: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1"
  2. There were no recent errors counted in the LogAttribute counter we set up to record errors;
  3. The Tasks/Time count in the PutSQL processor was though the roof, despite the errors and lack of successes.

Needless to say, the processor was all bound up and a number of tasks were queued.  Not a good start to my day.

I checked the data provenance and didn’t see anything remarkable about the backed-up data.  The error message suggests (to me) that the first statement parameter is at fault, and that parameter happened to be a date (which has been problematic for me in NiFi with a MySQL backend).  Neither that value, nor the rest of the values, were remarkable or illegal for the fields they’re going into.

It wasn’t until I spent some time looking over the source data that I saw the problem: there is a duplicate key in the data.  This error is NiFi’s way of complaining about it.

In our case the underlying table doesn’t have good keys, or a good structure in general, and I’m planning to replace it soon anyway, but updating the primary keys to allow the duplicate data (because it IS valid data, despite the table design) has solved the issue.

Komodo IDE Headaches

I’m slowly coming around to the idea that an IDE might be useful for PHP/Symfony projects (still not convinced about other languages and frameworks) and I’m currently trying out ActiveState’s Komodo IDE 10 on Linux.

It looks great but it’s… buggy.  One day in and I’m already getting frustrated with it.

  • The preference file doesn’t appear to be saved until the application is closed, if the application crashes it’s not clear that your changes will be saved.  This might be a safety feature, but probably not, because…
  • At least some preferences don’t take effect until the application is closed.  Not the ones that you’re warned about like checking remote files for changes, but other ones like ‘Allow file contents to override Tab and Indentation settings’ (which itself is unreliable since at least 2011).
  • When changing preferences, there is more than one place to change: Edit / Preferences, Edit / Current File Preferences, and Project / Project Preferences (the last is not under the Edit menu).
  • The cursor blinks by default (which is super annoying when you’re moving the cursor around the screen) and there isn’t an explicit option to disable it.  You have to create a custom JavaScript script that executes at every file open.
  • The toolbar icons are heavily styled, making their use opaque and the tooltips mandatory reading.
  • It has already crashed while closing — which, per the above, I’m doing a lot.

It’s not all bad, there are some really nice features:

  • Vi keybindings, so things like ‘A’ to start appending to the current line, or ‘/’ to search the current file, are really nice to someone who uses vim every day.
  • I do appreciate the ability to script things
  • The syntax highlighting and coloring seems more reliable (i.e. harder to confuse) than average.
  • The installation to a local directory was painless, and an icon properly shows up in the applications menu (I use Mate).  The default installation dir is to your home directory instead of /usr/local (which is the right thing to do for trial software, imho).

I want to like this editor, I really do, but it’s just going downhill as I work with it more.  At $250 per license it’s hard to justify the expense to my boss unless I really like it.

Raiser’s Edge, love, what were you thinking

TL;DR: In the SQL Configuration Manager, set the TCP port to a static one, if is trying to use Dynamic, and remove the Dynamic port. On the Server, go into the Advanced settings for the Firewall, and set Inbound rules for both the TCP and UDP ports, allowing them to connect. On the client, set Outbound rules in the Firewall manager for the same. Support Articles at bottom of post.

So, I do third party tech support for a couple of independent schools. Several of them use a program called Raiser’s Edge to keep track of charitable donations, and solicitations. This is all well and good, and the program certainly does the job, but sometimes it makes you want to down a liter of vodka and go home.

The set up: We had to replace a machine that was hosting a networked install of a Raiser’s Edge database. We didn’t realize it was networked until we got the call that they couldn’t access it from their laptop (crap).

The initial troubleshooting: First, we needed to uninstall RE from the laptop (the client computer). It would not go. Finally decided to reboot the machine to make sure RE was not running anywhere. Suddenly, the uninstall went like a breeze.

Now we needed to install RE from the network share on the server. We can’t connect. That took turning off all the firewalls on both machines to fix, but still, we could not get to the “Deploy” folder, that should have been the only available network share on the machine.

Turns out that the installer does not set that folder to be “Available” across the network. There was no documentation for that. Set it to “Available” and boom. I can see the network share.

Run the setup.

Install RE.

Try to run RE.

Start getting database errors. Native Error 17 – Can’t connect to the Database. Call Support and they say “Yeah, its probably the firewall.”

I was too irritated to tell them there was no firewall turned on, at first, but when I mentioned it, they said that it was possible an antivirus had blocked the ports needed. Go to this KB article and open the ports.

Yeah. Fine.

I go through the directions, figure out that SQL has a dynamic port, and follow the directions for that configuration. It doesn’t work. Fantastic.

Finally, Darling Husband o’Mine says, “Why don’t you specify the port it uses?”

So in the end, this is what worked:

  • In the SQL Configuration Manager, set the IPALL port to 1433
  • Stop and restart the SQL service/reboot the machine (I wound up rebooting, but YMMV).
  • On the Server, in Firewall Management, under Advanced Settings, set up Inbound Rules for the following:
    • TCP port 1433 open (or some open port)
    • UDP port 1434 open
  • On the Client, set up Outbound Rules for the following:
    • TCP port 1433 open (or whatever port you used on the server. THEY HAVE TO MATCH)
    • UDP port 1434 open
  • Install RE on the client machine from the Deploy share on the Server.
  • Test the connections.

If any of this makes no sense, here is the supporting documentation for all of it:

Good luck, and may the force be with you on this one.

Beautiful Code

The mark of a master programmer is someone who writes code that a novice can debug.
— attribution unknown

I read this quote, or something very similar to it, a long time time ago when I was just starting out.

I take the idea behind that quote to mean that master programmers have the experience to find the simplest solutions, which are easier to understand, but they also make their code easier to read so errors stand out.

It came back to me while reading a novice’s request for help in debugging something.  The example was a mess, with lots of extra activity, but it was also dense and poorly formatted.  The very simple bug was hard to see because of the sheer amount of code and the inconsistent formatting.

I strive to find simple solutions to the code I write, but I also strive to make my code neatly formatted and well-spaced.  I generally limit my lines to ~78 characters; I vertically align related operators; I leave space around operators.  This goes hand-in-hand with simple code: short functions that only do one thing; effective naming of things; do the least possible.  Together these generally make code that is both robust and easy to maintain.

I think of formatting to be like engineering a bridge.  Dense code is like big thick columns, steel plates, and stone architecture — it gets the job done, but it looks so heavy.  The best bridges are light and airy, full of empty space, yet they are stronger and more resilient.

PS: if you know this quote, and know who said it first, please drop me a line so that I can attribute it properly!

NiFi Build Error

I’m testing NiFi out on my local Gentoo installation to prepare for an implementation at work, and after a rather lengthy build/test process (“ten minutes” my fanny) ran into this error:

$ mvn clean install
[INFO] Scanning for projects...
...
'Script Engine' validated against 'ECMAScript' is invalid because Given value not found in allowed set 'Groovy, lua, python, ruby'

This error left me scratching my head.  Nothing related to JavaScript/ECMAScript dependencies were mentioned anywhere.  How would you get it, anyway?  Webkit, I suppose…

Sudden epiphany: this is a new Gentoo installation, and this program, including the build script, is running Java.  Gentoo doesn’t install Sun Oracle’s Java by default, but instead comes with IcedTea out of the box.  It’s acceptable for some simple uses, but is buggy for any complex. (Minecraft is a great example where it just doesn’t work.)  I haven’t used Java for anything yet, so I hadn’t installed the JDK yet.  The build instructions specify JDK 1.7 or higher, but I didn’t think anything of it because I’m used to just having it installed.

echo "dev-java/oracle-jdk-bin Oracle-BCLA-JavaSE" \
  >> /etc/portage/package.license/file
emerge -av dev-java/oracle-jdk-bin
...
$ mvn clean install
[INFO] Scanning for projects...
...
[INFO] BUILD SUCCESS

Finally!