MarkLogic NoSQL

I recently attended a few training sessions for MarkLogic held at an office in a nearby business center. Now, I’ll forgive you for not knowing what MarkLogic is, as even I hadn’t heard of it before six months ago. MarkLogic is (apparently) the leading Enterprise NoSQL provider.
 
NoSQL is big and sexy right now because of the supposed advantages in handling big data, and large web companies like Google and Facebook use a lot of NoSQL in the backend. Most of the popular/well-known NoSQL solutions are open-source/free ones: MongoDB, Cassandra, CouchDB, and so on. But these aren’t actually very popular on the enterprise side, hence “Enterprise NoSQL” isn’t a very common phrase.
 
One of the reasons NoSQL isn’t currently very popular for enterprise projects is that the popular open-source solutions such as MongoDB don’t guarantee ACID transactions. In fact, MongoDB has the concept of “eventual consistency” for their distributed servers, which implies that they don’t have real-time consistency.
 
MarkLogic does guarantee ACID transactions, along with government-grade security. Both of these are things that enterprise clients love. So that’s the market they’re in. Their highest-profile project to-date was the backend datastore for Healthcare.gov (also known as Obamacare/ACA). That project involved consolidating data with different structures from multiple sources and needed to scale up to a ridiculously high capacity, so it seemed tailor-fit for a big data level NoSQL solution.
 
During the second-to-last day of the training, Jason Hunter, CTO of Marklogic for Asia-Pacific, dropped in to answer some questions. He talked a bit about how MarkLogic started and how they got him on board, and a bit about their competition (some disparagement towards MongoDB and Oracle) and about Healthcare.gov.
 
He also had a really good sales pitch about why NoSQL is just a better approach compared to RDBMS. (Although it would have been more fun if he had given this talk with like an Oracle salesperson there to debate with him.)
 
One of his points was that RDBMS restrictions such as limits on column size are outdated. They were sensible in the old days when disk space was at a premium, but these days you don’t need to limit how many characters you store in a last name field. MarkLogic stores all records as documents with no file size limit (AFAIK) to avoid such an issue. From experience developing web forms, there’s the occasional client or system user who doesn’t understand why we need to have character limits on fields. We also had this system where they wanted to be able to type unlimited-length rich text content (we stored it as HTML in the backend) in memoboxes. These were fields where they had to write stuff like notes and assessments and most of their data was pages of text. I feel like that sort of thing would have been a great use for NoSQL.
 
Another point he raised was that RDBMS systems were very bad at allowing the user to search like three columns at random. You needed to know in advance which three columns to index. For document stores like MarkLogic, typically the entire document is indexed so that problem is avoided. Of course, that means that functions like “sorting by last name” need a bit more setup. You need to build a range index in MarkLogic to sort by a specific field. So it’s kind of a trade-off either way.
 
Now to be fair, Oracle does support indexed full-text search over any number of columns via Oracle Text. But it’s not the default behavior and definitely not straightforward. I used to work with Oracle Text a lot in some of my older projects, and the amount of time it took to index any nontrivial amount of data often gave us a headache.
 
I should do a test sometime to determine how well MarkLogic’s indexing performs. The story goes that MarkLogic started out as a document search application before they changed gears to become a database. They even approached VC’s with the intent of competing with Google. They’ve had a lot of time to get good at this, so I have high expectations.
 
The MarkLogic server itself is an interesting piece of engineering. It’s basically a document store, a search engine and an application server all rolled into one package. Upon installation you get some administrative web applications for configuration purposes. The admin interface seems robust and thorough. Contrast that with Oracle where you often find yourself needing to tinker around with configuration files and such
 
You can run web applications on the MarkLogic server itself. The supported languages are XQuery and server-side JavaScript. Odd choices I know. I suspect due to historical reasons they started out with XQuery, but the SJS side has the same capabilities (or so we’re told). If you’re not a fan of either option, you can also expose the server’s functionality via a REST interface. They also provide existing Java and Node.JS APIs on top of that REST interface. All of this means you can deploy any kind of webapp in front of MarkLogic server.
 
The world is moving towards bigger data stores, so it’s not unreasonable to think that NoSQL is on the way up and will be big players in the future. So I think the training was worth it (even if I did have to stay in Ortigas for a while). It’s early still. MarkLogic might still turn out to be as painful to work with as Oracle was. But at the very least it’s interesting to try a different approach to enterprise data storage. Looking forward to see what kind of applications we can build with this tech.

Paella

A couple of years ago, two friends and I were being tourists in Barcelona. With its wide, spacious streets and strangely uniform city blocks, we walked around a lot. During one of our tourist days, we decided to eat some paella on the way back to our AirBNB. Who comes to Barcelona and doesn’t eat paella right?

We ended up walking for quite a while. Every time we came upon a new restaurant that served paella we would consider the price and the restaurant and would think, hey maybe we can find somewhere better or cheaper further along the way. At one point a Pinoy working in a nearby restaurant even overheard us talking about it in Tagalog and he called after us as we walked away. “Dito, mura lang!”, but we paid him no heed.

How could we know which restaurant was the best one until we walked all the way and saw all the options? But walking all the way back to the AirBNB to see all the options meant we would go for a longer time tired and hungry, so that was obviously no good. Even worse, we may find at the end of the walk that the best choice was too far back and we end up with a suboptimal paella. At some point, we would have to decide that “Okay, the paella here looks good enough, let’s eat here.”

I thought it was kind of a metaphor for life and choices and such. You can’t know ahead of time whether the options presented to you now are the best you’ll ever get. But at some point you have to stop browsing and overthinking and decide to settle for something that is “good enough”. Sometimes it turns out great, sometimes it ends poorly, but it’s almost always a gamble. Life decisions are only hard if we make them hard. But as long as you made the best decision with your limited information, I think you can always choose to live without regret.

We ended up eating at this small diner near our AirBNB. It was run by an Asian dude and his family, though it seemed they spoke better Spanish than they did Chinese or whatever. I had the Valencian paella which had meat instead of seafood or whatever (no surprise for anyone who knows my dietary habits.) I don’t know whether it was the best or the cheapest paella we could have eaten that day, but I could tell you it was pretty damn good anyway.

Review: Final Fantasy XV

With The FFXIII trilogy not being particularly well-received and FFXIV being an MMO, Final Fantasy XV has been a long awaited as the next mainline single-player game in the much-acclaimed series. This review will have minor spoilers.

Story

FFXV follows the story of Noctis, prince of Lucis and his band of brothers (okay they’re not really brothers, but they might as well be). They’re supposed to be on a road trip to get Noctis married, but things happen along the way and eventually they have to figure out how to liberate their homeland from The Evil Empire. That’s the backstory.

The story as presented in the game has a few problems. There’s a lot of significant back story and goings-on that happen off-camera:

  • There’s both a feature-length movie (Kingsglaive) and a six-episode anime (Brotherhood) that expounds on the four main characters’ backgrounds and how the current political situation got to where it is
  • There’s supposedly a whole effort by minor characters like Cor to organize a resistance against the Empire, but we’re never really exposed to it
  • Then there’s Lunafreya, Noctis’ betrothed, who goes off and does her own stuff to try to undermine the Empire and supposedly help Noctis, but for most of the game it’s never really explained what exactly she’s doing or how it’s relevant
  • One of your friends, Gladio, goes off on a short sidequest of his own and vanishes for a while, and it’s never explained WTH that was about. Presumably it will be covered in a future DLC.

Another problem is that Noctis and the gang spend most of the game sort of lollygagging around enjoying their road trip without much sense of urgency. Understandably they wanted to present an open world this time (since much of the criticism of FFXIII was for its linearity), but the problem is that the open world is filled with a lot of frivolous things that don’t seem particularly urgent given the fate of the world being at stake. I mean, the party spends a lot of time camping out, cooking food, fishing, taking pictures, driving around looking for auto parts, and so on. The sense of urgency is not apparent.

I suppose the lack of urgency is an acceptable trade-off for the open world, but even for the supposed main story quests in the earlier chapters, it’s not immediately clear how they serve to help you overcome the Empire. I feel like there should have been more story quests related to helping establish a resistance and so on.

The later chapters narrow down the open world significantly, and the second half of the game takes place outside the main continent where there is significantly less freedom to explore. It feels a bit like they spent a lot of their design budget on the open world half and so just restricted the second half to the meaty story parts. Reminds me a bit of Xenogears disc 2!

One of the chapters near the end was also executed very poorly. To establish a sense of tension, most of the systems you had been relying on throughout the game are taken away from you. While this would have been okay for a short segment, the chapter goes on a bit too long and ends up being far too tedious.

I did find myself surprised by events leading up to the final chapter, although I felt like it was another wasted opportunity. The final chapter gives us a glimpse at a different version of the open world but never really let us see most of it.

The ending itself was bittersweet (and easily trolled haha), but I found it an acceptable ending.

Characters

While I found the main cast of Noctis, Ignis, Gladio and Prompto to have engaging personalities, they don’t really have much backstory in-game (most of the character backstories are in Brotherhood). In fact, later on, a startling revelation is made about one of them and none of them really bat an eye at all. It was in the sense of “hey, we don’t care about that, we’re your friends,” but it still felt like something that should have been explored further.

The interaction among the group is great though. They regularly banter and tease each other even during combat. It really felt like they were a group of long-time friends.

Combat 

I’m an old-school FF fan, so I’ll be the first to admit I miss ye olde active time battle. FFXV’s battles take place entirely in real-time, with the player controlling Noctis and the other characters controlled by AI. The AI control goes about as well as you could expect, they do fine against weak enemies but against tougher battles you find yourself constantly having to help them with potions or other items. You can command them to do special attacks when available, and they will participate in combo attacks when possible, but for the most part it’s just Noctis warping and striking.

Combat is fluid and there’s a lot of movement and jumping around and looking for opportunities to strike from behind. Early on you are discouraged from wandering around at night as the enemies might be too high level, but the combat system is set up such that most tough battles can be overcome simply by having enough recovery items around. I only got a game over once, early in the game when didn’t know what I was doing and I tried wandering at night and promptly got squashed by an iron giant.

The Open World

The open world gives the party a lot to do. There’s “hunts” dished out by people in diners (basically quests to go kill a specific bunch of monsters) and other side quests given out by various characters. There’s a fishing minigame, because apparently it’s a hobby of Noctis. The fishing minigame is okay, except I didn’t have the patience to try to get the most difficult fish.

Then there’s the photos and the food porn. A lot of pandering to the modern-day youths here, I’m surprised instagram hasn’t been invented in their world. Cooking is Ignis’ specialty, and the game renders each dish really well, sometimes enough to make you hungry.

And the photos are the specialty of Prompto. It’s one of those things that feels really frivolous in the game. Every time you camp you can browse pictures that Prompto has taken so you can save them for later viewing. A later patch provided a photo mode that you can control directly. There’s an entire reddit thread about how players have snapped more pics for this game than for their own vacations!

Anyway, I’m no exception. Exporting a lot of photos is a pain though, as there is no batch export option in-game. Here are a sampling of the photos I got (a bunch of them are from achievements):

View post on imgur.com

Achievements

Speaking of achievements, I was a touch disappointed with FFXV’s achievement set. I got the Platinum trophy, but I felt like it was a bit too easy, as there was still a lot of things to do in the game outside of the achievements. I don’t want annoying grindy achievements, but I don’t want them to be too easy and “no challenge” either.

Other stuff

They had this weird Chocobo carnival DLC that somehow takes place in another timeline or something LOL. IDK how it works.

Despite the fact that the more action-oriented combat system is indicative of where SE plans to take the series moving forward, there’s still a significant amount of nostalgia in the game for old-schoolers to appreciate. Callouts range from Prompto humming the classic victory fanfare after the battle to 2d sprites in the store menus jumping up in down to indicate which characters can equip each item. Classic FF beasties like the Malboro still make an appearance (and are still a pain to fight!)

The weirdest thing overall has to be the Cup Noodles thing! They had some sort of tie-up with Nissin and Cup Noodles is one of the meals your party can have in the game and Gladio is obsessed with them. Later on you even get a quest to try to find a way to improve on Cup Noodles and it ends with the characters saying there’s really no beating the original since it has a perfect mix of ingredients already. I feel like the VA’s really enjoyed doing those lines haha.

Overall

Okay, I had a bunch of complaints and nitpicks but to be honest, I enjoyed the game. There was a lot of gameplay, some seriously though battles and some annoying dungeons that took forever, but I finished most of it and got the platinum in under a month. Nowadays for an RPG that’s very quick for me, an indication of how much I liked the game. There’s a bunch more content planned for it, including DLC story packs for each of the other main characters, and high-end boss fights and what not. Not sure if I’d still play those since there’s a lot more RPGs on the horizon (2017 is a good year for RPG gamers.) But if Final Fantasy XV is the next step for the series moving forward, I find it acceptable.

Google Code Jam and Competitive Programming

Back in 2004, I signed up for the Google Code Jam for the first time. Unfortunately I didn’t make it past the qualifying round.

I was a bit luckier in 2008 and 2010, making it to round 2 both times. In fact in 2008 as I recall I was one of only two participants from the Philippines who made it to round 2, which allowed me to jokingly brag about being the #2 programmer in the country.

I registered again a few times after that, and tried again last year (2016), where I only made it to round 1.

I’m no stranger to programming competitions, although the GCJ was the only one I’ve ever really participated in. Back in college, one of my friends usually joined such competitions, and I had a good time going over some of the problems with him.

Programming competitions don’t usually reflect the kind of programming done in most real-world jobs (although it might be interesting to use competitive programming problems in whiteboard interviews and see what happens). They focus on pushing the boundaries of the programmers talents in problem solving. You also need to be fairly familiar with algorithmic approaches to searching, sorting, and so on. Having a decent background in math also helps a lot.

The Google Code Jam in particular focuses on scaling up your code to handle very large amounts of data – well past the integer limit sometimes. Each question usually has a small and a large input. For the large inputs, you only have a short amount of time (I think it was 5 minutes?) to submit the output. Presumably you wrote and tested your code before starting the timer, so the short time is really only for your code’s execution. You can afford to be inefficient or maybe even brute force your way through the small input, but if you want to solve the large inputs on time you have to be able to optimize your approach.

When I first signed up, I dreamed of winning the entire thing (and of course being able to brag about being the best programmer in the world). But realistically, a lot of the people joining these competitions are “pros” more or less, so the chance of a “casual” like me winning are not that high. They are “pros” not in the sense that they do it for a living, but more of they do a lot of these competitions and regularly practice for them. Websites like TopCoder provide such competitions regularly for those who are interested. Also, I feel like I am very old and rusty, having worked too long in web development stuff where my focus is usually on a different kind of problem.

Still, I think it’s a good idea for programmers of all skill levels to give it a try, just to see where you are at. Who knows, you might even make it all the way! The next Google Code Jam starts soon – the qualification round is on Apr 7. I signed up, although I’m unsure if I will have the time this year even if I advance, due to scheduling problems. But a man can still dream, right? Who knows, maybe I’ll see some of you there too in the final round!

Scary Movies and Games

I’m not a fan of scary movies. I don’t appreciate the idea of paying money to get surprised by jump scares or whatever.

Back when I was a kid I remember my dad watching a Betamax copy of The Gate back home and me and my younger brother were watching with him and the movie seriously creeped me out. There was this one scene where a demonic eye manifested on the lead kid’s palm and that scene stuck with me for a while. (Maybe some latent form of trypophobia – don’t google it.)

Slasher movies I find a bit more acceptable – I watched a few of them in the 90s like Scream and I Know What You Did Last Summer. But the really creepy stuff like The Ring and similar films I’m not particularly interested in.

For scary video games, we played a few of them back on the PC when we were young. The most memorable would definitely be Waxworks, a kind of point-and-click with first-person grid-exploration horror game. I remember my brother and I would play it late at night while the folks were out. There was one time we had to stop because we had been surprised at the sudden appearance of some bad guy in the pyramid stage. We had to turn off the PC and run out to the sari-sari store so that we had some people to talk to. We eventually managed to finish this game, but I think after that part we refrained from playing too late in the night. LOL.

I also played the original jump scare game, Alone in the Dark although I never got into the succeeding series like Resident Evil. These days I don’t bother with scary games anymore. Sometimes I consider playing stuff like Amnesia, but I’m sure even if I’m a grown-up I’d still get creeped out by some of the stuff and that’s not really how I like to be entertained.

 

 

Fake Devs and Whiteboard Interviews

Recently, a developer needed to undergo a tech interview at US immigration:

This may surprise some people I’ve worked with, but I didn’t have formal computer science training in school. I’m not actually a computer science major. Yet I’ve worked as a software developer for more than a decade now. Literally zero times have I needed to write a sorting function or balance a BST.

I have a rudimentary understanding of some sorting algorithms (mostly just bubble sort and selection sort), and I have some idea of how to balance a BST. And if I needed to implement any of those I could probably whip something up after a few tries. (Something buggy and/or inefficient probably.) If I had to interview at a company that used one of these whiteboard interviews, there’s a good chance I wouldn’t pass.

For maybe 90% (or even as high as 99%) of modern-day programming work, it simply isn’t needed to know these things by heart. Those skills are more necessary if you’re doing low level things like writing an OS or a graphics driver or such. For everyday web development or Java or Rails or whatever, it’s an indulgence. And even for those instances that you need them, well, the internet is right there.

I’m not a fan of the idea that not knowing data structures and algorithms means you aren’t a real software developer. I have a couple of developer friends who jokingly refer to themselves as “fake devs”. It’s supposed to mean someone who doesn’t have any formal computer science background and has at best a passing familiarity with data structures, algorithms, and other computer science concepts. As one of them stated “I wouldn’t know how to make a map without using a map.” (I thought about writing a post about that, but it’s surprisingly easy – just look it up.). One of them even asked me to give them a quick crash course in data structures and algorithms because he was applying in a large and well-known software company.

For me, this term is silly. If you’re doing software development work, then you’re a software developer.

I’m not saying knowledge of these things isn’t useful. If you want to do more complicated work it is certainly useful to have these tools in your toolbox. Knowing them also makes you more flexible and expands your thinking. And certainly one should be the type of developer who is willing to seek out that knowledge if it turns out you need to use it.

But the problems with the whiteboard interviewing process isn’t that they require knowledge of these things; it’s that they require it under pressure, off the top of your head, without access to the internet. This doesn’t reflect how programming work happens in the real world, where you often have a team, an IDE, and even the entire internet to help you.

Unfortunately, it’s often far too costly for companies to evaluate you under real-world conditions, so many resort to this common interviewing process. After all, it is difficult to evaluate the performance of even developers you have been working with for months, what more someone who just whipped up a short function on a whiteboard in twenty minutes? We tried some variations on it at our last company, but for me it is a problem yet unsolved. Maybe someday…

 

Sony 4K HDR Bravia X8000D

4k XLocal tech blog YugaTech is doing a giveaway of a Sony X8000D Bravia 4K TV. To be honest I wasn’t sure about using this blog to participate in a promo, but I was already in the market for a new TV since our living room TV is already starting to have some problems.

I checked out the product features of the Sony X8000D Bravia on their website. Some of the features stood out for me specifically:

  • 4K HDR – this means you get to watch high definition 4k videos. Perfect for those who really enjoy high quality displays. The TV also comes with some technologies to enhance your viewing experience such as 4k X-Reality and Triluminos display. And if you have a gamer in the family, this means you can now maximize the Playstation 4 Pro since it supports gaming in 4K resolution!
  • Connectivity – Wifi is a big deal for me because our current TV is a bit old and only supports wired connection. Wifi makes it a lot easier! The TV also has 4 HDMI slots – let me see… that’s one each for the PS4 Pro, Nintendo Switch, and XBox One, plus one more to spare for those times when you want to connect your laptop to a big display (one can dream right?)
  • Android TV – this means all the apps on the Google Play store can be downloaded and run on the TV. That means built-in Youtube, Netflix, etc, without need for a separate device to stream from!
  • Google Cast – oh, I didn’t know this even existed until now! Apparently TV’s can now have Chromecast functionality built-in, without the need for the separate Chromecast device. Awesome! We can just send content from our mobile devices directly to the TV!
  • Netflix recommended – one of the things I’ve been looking for in a new TV is that I want my folks to be able to easily use my Netflix account. Currently the only way they can use it on the TV is through the PS4, which is a bit difficult if you don’t know your way around a controller. Having the TV itself support Netflix will make it a lot easier for my parents to take advantage of Netflix’s library of movies and TV shows!
  • X-Protection PRO – to protect against power surges, this seems like it would be really good especially here in Metro Manila where power supply is not always the most stable. Added protection for your investment!

The model has a lot of other features designed to enhance your enjoyment of audio and video content. With all of these features, the Sony Bravia X8000D could definitely be the perfect home entertainment companion for the modern family!

Planning / Reacting / Relaxing

At any point in time, what you are doing can be grouped into one of four buckets:

  1. Planning
  2. Executing a plan
  3. Reacting to something
  4. Relaxing (leisure time)

Overthinkers tend to do too much of #1. The most efficient people probably spend most of their time in #2. People whose lives are chaotic do too much of #3. Almost all people don’t get enough #4. (I might be doing too much of #4~)

Running Python 2.x and 3.x on Windows

I’ve been hesitant to try Python 3.x because it’s not backward compatible with Python 2.x which I’ve been using for scripting since forever. But recently I found out that since Python 3.3, they’ve included a launcher in the Windows version that supports having both versions installed.

You can use the launcher to specify the Python version to use at the command line (it defaults to whichever version was installed first):

Even better, the launcher will recognize a header in your .py files that can specify which version of python to use:

#!/usr/bin/env python3

If the launcher sees this header, it will automatically launch the appropriate python version. Handy!

Boxes

We put people into boxes because it is convenient. It’s easier for our mental model of the world to say to yourself things like “This guy works with computers, maybe he can tell me how to fix my printer.” or “This person is from [school] and they are very arrogant.” or “You’re from [country]? You guys do [that country’s thing] right?” or  “This person is a supporter of [politician] so he must support all the things that politician does, even the things I hate.” Or in the modern world, there are even worse stereotypes.

Boxes and stereotypes are convenient because it means we don’t need to relearn our reactions and opinions when presented with a new person. We simply choose a convenient box to put them in and then we convince ourselves we are treating them appropriately.

But the real world is not so discrete. The real world is a lot more granular, a lot more nuanced. While it is inconvenient for your mental model, people will usually not fit exactly into any of your preconceived boxes. Or they may fit in multiple boxes. Or maybe none at all.

Every time you put a person into a box, you discredit her by some amount. In your mind, you shave off some of who he is a person so that he better fits your box. And for many instances, maybe that’s okay. Maybe you don’t have any significant interaction with that person, and the simpler mental model is sufficient. You don’t need to understand the character nuances of every grocery store cashier for example.

But beyond that, in order to pursue any kind of meaningful interaction with another person, that is insufficient. We must be willing to set aside whatever box you have put a person into and be willing to understand all those little bits we set aside. Those nuances characterize and define the individual, not whatever box we have put him into.