First they came for the verbs, and I said nothing because verbing weirds language. Then they arrival for the nouns, and I speech nothing because I no verbs.
Source: http://www.bash.org/?954654
“ERROR: … failed to process… ” in NiFi
I was greeted by a few cryptic things in NiFi this morning during my morning check-in.
- A PutSQL processor was reporting an error:
"ERROR: PutSQL[id=$UUID>]failed to process due to java.lang.IndexOutOfBoundsException: Index: 1, Size: 1; rolling back session: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1"
- There were no recent errors counted in the LogAttribute counter we set up to record errors;
- The Tasks/Time count in the PutSQL processor was though the roof, despite the errors and lack of successes.
Needless to say, the processor was all bound up and a number of tasks were queued. Not a good start to my day.
I checked the data provenance and didn’t see anything remarkable about the backed-up data. The error message suggests (to me) that the first statement parameter is at fault, and that parameter happened to be a date (which has been problematic for me in NiFi with a MySQL backend). Neither that value, nor the rest of the values, were remarkable or illegal for the fields they’re going into.
It wasn’t until I spent some time looking over the source data that I saw the problem: there is a duplicate key in the data. This error is NiFi’s way of complaining about it.
In our case the underlying table doesn’t have good keys, or a good structure in general, and I’m planning to replace it soon anyway, but updating the primary keys to allow the duplicate data (because it IS valid data, despite the table design) has solved the issue.
Komodo IDE Headaches
I’m slowly coming around to the idea that an IDE might be useful for PHP/Symfony projects (still not convinced about other languages and frameworks) and I’m currently trying out ActiveState’s Komodo IDE 10 on Linux.
It looks great but it’s… buggy. One day in and I’m already getting frustrated with it.
- The preference file doesn’t appear to be saved until the application is closed, if the application crashes it’s not clear that your changes will be saved. This might be a safety feature, but probably not, because…
- At least some preferences don’t take effect until the application is closed. Not the ones that you’re warned about like checking remote files for changes, but other ones like ‘Allow file contents to override Tab and Indentation settings’ (which itself is unreliable since at least 2011).
- When changing preferences, there is more than one place to change: Edit / Preferences, Edit / Current File Preferences, and Project / Project Preferences (the last is not under the Edit menu).
- The cursor blinks by default (which is super annoying when you’re moving the cursor around the screen) and there isn’t an explicit option to disable it. You have to create a custom JavaScript script that executes at every file open.
- The toolbar icons are heavily styled, making their use opaque and the tooltips mandatory reading.
- It has already crashed while closing — which, per the above, I’m doing a lot.
It’s not all bad, there are some really nice features:
- Vi keybindings, so things like ‘A’ to start appending to the current line, or ‘/’ to search the current file, are really nice to someone who uses vim every day.
- I do appreciate the ability to script things
- The syntax highlighting and coloring seems more reliable (i.e. harder to confuse) than average.
- The installation to a local directory was painless, and an icon properly shows up in the applications menu (I use Mate). The default installation dir is to your home directory instead of /usr/local (which is the right thing to do for trial software, imho).
I want to like this editor, I really do, but it’s just going downhill as I work with it more. At $250 per license it’s hard to justify the expense to my boss unless I really like it.
Raiser’s Edge, love, what were you thinking
TL;DR: In the SQL Configuration Manager, set the TCP port to a static one, if is trying to use Dynamic, and remove the Dynamic port. On the Server, go into the Advanced settings for the Firewall, and set Inbound rules for both the TCP and UDP ports, allowing them to connect. On the client, set Outbound rules in the Firewall manager for the same. Support Articles at bottom of post.
So, I do third party tech support for a couple of independent schools. Several of them use a program called Raiser’s Edge to keep track of charitable donations, and solicitations. This is all well and good, and the program certainly does the job, but sometimes it makes you want to down a liter of vodka and go home.
The set up: We had to replace a machine that was hosting a networked install of a Raiser’s Edge database. We didn’t realize it was networked until we got the call that they couldn’t access it from their laptop (crap).
The initial troubleshooting: First, we needed to uninstall RE from the laptop (the client computer). It would not go. Finally decided to reboot the machine to make sure RE was not running anywhere. Suddenly, the uninstall went like a breeze.
Now we needed to install RE from the network share on the server. We can’t connect. That took turning off all the firewalls on both machines to fix, but still, we could not get to the “Deploy” folder, that should have been the only available network share on the machine.
Turns out that the installer does not set that folder to be “Available” across the network. There was no documentation for that. Set it to “Available” and boom. I can see the network share.
Run the setup.
Install RE.
Try to run RE.
Start getting database errors. Native Error 17 – Can’t connect to the Database. Call Support and they say “Yeah, its probably the firewall.”
I was too irritated to tell them there was no firewall turned on, at first, but when I mentioned it, they said that it was possible an antivirus had blocked the ports needed. Go to this KB article and open the ports.
Yeah. Fine.
I go through the directions, figure out that SQL has a dynamic port, and follow the directions for that configuration. It doesn’t work. Fantastic.
Finally, Darling Husband o’Mine says, “Why don’t you specify the port it uses?”
So in the end, this is what worked:
- In the SQL Configuration Manager, set the IPALL port to 1433
- Stop and restart the SQL service/reboot the machine (I wound up rebooting, but YMMV).
- On the Server, in Firewall Management, under Advanced Settings, set up Inbound Rules for the following:
- TCP port 1433 open (or some open port)
- UDP port 1434 open
- On the Client, set up Outbound Rules for the following:
- TCP port 1433 open (or whatever port you used on the server. THEY HAVE TO MATCH)
- UDP port 1434 open
- Install RE on the client machine from the Deploy share on the Server.
- Test the connections.
If any of this makes no sense, here is the supporting documentation for all of it:
- Native Error 17 KB article from BlackBaud (great resource, by the way): https://kb.blackbaud.com/articles/Article/42647
- How to figure out what Port your SQL Server is using: https://kb.blackbaud.com/articles/Article/39342
- Configure a Server to listen on a specific port in the SQL Configuration Manager: https://msdn.microsoft.com/en-us/library/ms177440.aspx
- How to modify the Windows Firewall for SQL Database Engine Access: https://msdn.microsoft.com/en-us/library/ms175043.aspx?f=255&MSPPError=-2147217396
Good luck, and may the force be with you on this one.
Beautiful Code
The mark of a master programmer is someone who writes code that a novice can debug.
— attribution unknown
I read this quote, or something very similar to it, a long time time ago when I was just starting out.
I take the idea behind that quote to mean that master programmers have the experience to find the simplest solutions, which are easier to understand, but they also make their code easier to read so errors stand out.
It came back to me while reading a novice’s request for help in debugging something. The example was a mess, with lots of extra activity, but it was also dense and poorly formatted. The very simple bug was hard to see because of the sheer amount of code and the inconsistent formatting.
I strive to find simple solutions to the code I write, but I also strive to make my code neatly formatted and well-spaced. I generally limit my lines to ~78 characters; I vertically align related operators; I leave space around operators. This goes hand-in-hand with simple code: short functions that only do one thing; effective naming of things; do the least possible. Together these generally make code that is both robust and easy to maintain.
I think of formatting to be like engineering a bridge. Dense code is like big thick columns, steel plates, and stone architecture — it gets the job done, but it looks so heavy. The best bridges are light and airy, full of empty space, yet they are stronger and more resilient.
PS: if you know this quote, and know who said it first, please drop me a line so that I can attribute it properly!
NiFi Build Error
I’m testing NiFi out on my local Gentoo installation to prepare for an implementation at work, and after a rather lengthy build/test process (“ten minutes” my fanny) ran into this error:
$ mvn clean install
[INFO] Scanning for projects...
...
'Script Engine' validated against 'ECMAScript' is invalid because Given value not found in allowed set 'Groovy, lua, python, ruby'
This error left me scratching my head. Nothing related to JavaScript/ECMAScript dependencies were mentioned anywhere. How would you get it, anyway? Webkit, I suppose…
Sudden epiphany: this is a new Gentoo installation, and this program, including the build script, is running Java. Gentoo doesn’t install Sun Oracle’s Java by default, but instead comes with IcedTea out of the box. It’s acceptable for some simple uses, but is buggy for any complex. (Minecraft is a great example where it just doesn’t work.) I haven’t used Java for anything yet, so I hadn’t installed the JDK yet. The build instructions specify JDK 1.7 or higher, but I didn’t think anything of it because I’m used to just having it installed.
echo "dev-java/oracle-jdk-bin Oracle-BCLA-JavaSE" \
>> /etc/portage/package.license/file
emerge -av dev-java/oracle-jdk-bin
...
$ mvn clean install
[INFO] Scanning for projects...
...
[INFO] BUILD SUCCESS
Finally!
New Job!
So ends one chapter of my life, and another begins. I begin my new job on Monday!
EIG Career
05/29/2012 – 03/25/2016
R.I.P
Bad Werdz
Swearing makes talking fun!
— Maureen Paul, EIG
NYC 2016
Meghan and I like to keep our kids exposed to the cultural world, giving them a cosmopolitan worldview. I chose a job near Boston in part to allow frequent trips into the city. But one city isn’t enough to give them a breadth of experience.
We took an overnight trip to New York City during February school vacation. Discovery Times Square is currently exhibiting The Vikings, which is of particular interest to Meghan and Alpha. (That isn’t to say there isn’t interest from me and Beta, they’re just that much more keen.) Meghan found a great deal on motel rooms (the girls are old enough now that we need two) right by Times Square at Four Points by Sheraton. The view wasn’t anything to write home about but the location can’t be beat: two blocks from Times Square. The rooms were clean and neat, and the staff were bend-over-backwards friendly. I would totally stay there again.
We drove down early Wednesday morning, dropping Butter off at Marty’s and picking up Meghan’s mom (aka Baba) on the way. Baba loves going to NYC, she knows the lay of the land better than we do, and she’s a bit more adventurous than I am when the kids are around, so we invited her along. She offered to share the kids’s room to keep costs down — and that provided peace of mind overnight, too.
We rolled into town around noon and got situated in the hotel with time to spare before our exhibit reservations. We walked from the hotel and poked around Times Square for a few minutes.
There were a pair of people in costume, one dressed as Woody from Toy Story and the other as Olaf from Frozen, soliciting tips. They made their own costumes and don’t work for anyone but themselves, so Meghan got photos with each and tipped them a few bucks.
I continue to be amazed how many people are continually in New York City. The crush of people feel like there’s an event going on somewhere, but it’s really just an every day occurrence. Dozens of people at every crosswalk, hundreds of people on every sidewalk, all the time. Boston has nothing on NYC.
We made our way through the crowds to the exhibit and spent about an hour and a half learning about Vikings. As an aside: the exhibit is self-paced and just about the right length. They had a number of artifacts grouped into several themes about everyday life, instead of constructing a sequential historical narrative.
We waited to get lunch until after the exhibit so we were somewhat famished. A few storefronts down from Discovery is a pizza and Italian restaurant called John’s of Times Square, located in a former church. The adults had excellent pizza and the girls had excellent pasta. It might have been a case of hunger making the best sauce, but probably not.
After lunch we shopped around Times Square a bit. Alpha bought a New York-emblazoned sweat shirt and Beta got a pin for her hat. Baba noticed that Phantom of the Opera was playing right in front of us, so she bought three tickets for 8 o’clock that night. Beta wasn’t interested and I was wiped from driving all day; we chose to stay in.
We wandered back to the hotel to rest up. I had to step out in search of a pharmacy: we had a snafu when we left the house and forgot to pack our bathroom stuff.
Our motel offers complimentary dinner stuffs on Wednesday nights. We were just coming off of lunch, not particularly hungry, but the food was delicious. Free wine and beer, too!
We finished dinner and Beta and I headed up to our room to watch some TV and get ready for bed. Meghan, Baba, and Alpha headed out to their show, and didn’t get back until after I was asleep — sometime after 10:30 pm. They said they had a great time though!
We started the next day with an excellent breakfast buffet at the motel. We seem to have a knack for choosing motels with great breakfasts; it really makes dollars stretch further if your breakfast is hearty when you’re traveling.
We decided that we could fit in one more museum visit before leaving town. The American Museum of Natural History was on our way home and is always worth a visit. We also considered the USS Intrepid but decided to save it for another visit so we can give it the amount of time it deserves.
We got in early and encountered almost no waiting to get into the parking garage and admissions line. We had three destinations in mind: the dinosaurs, the blue whale, and the gift shop. We got to see all three, in that order. The life-size blue whale model is stupendously large – and that’s after seeing the full-size apatosaurus for comparison.
We made our way out of the city around 1 pm. We didn’t want to get home too late, and the drive can take upwards of six hours with stops and detours to Willimantic and Hampton. The ride home was uneventful, traffic was mostly light and Waze didn’t let us down. Butter the dog was very happy to see us when we picked her up, and I was extremely happy to sleep in my own bed.
My new camera lens
Yesterday was spent at the Boston Museum of Science with the Mystic Joneses. Today, we took Beta back for the thing she loves most: The Butterfly Garden. (We couldn’t go yesterday because it was sold out. Pro-tip, avoid the MOS on Federal Holidays at all costs.)
For my birthday I got something I’ve been wanting for a while: a set of lenses for my phone. I went a little nuts with the macro lens. I can’t even say I’m sorry about it.
Most of these were taken from just a couple of inches away. The butterflies and moths in this exhibit are so relaxed around people, its is amazing.
I played with my settings, and found out that trying to use the macro lens on butterflies on swaying branches was a guaranteed failure.
The details I managed to get were astounding, though.
Some pictures came out with some real drama. You can almost see his scales!
I am really pleased with these little lenses.
And of course, I had to do a gratuitous, super-close-up of a flower. For Reasons.
