Roy Tang

Programmer, engineer, scientist, critic, gamer, dreamer, and kid-at-heart.

Blog Photos Links Archives About

All entries tagged software development.

You can subscribe to an RSS feed of this list.

Nov 2019

I was helping my brother check some database issues the other week, and he mentioned how impressed he was with how quickly I was able to come up with SQL queries on the fly. I told him that SQL was one of the skills I considered myself to have mastery over. This shouldn’t be surprising given my early career path: for most of my first year working as a software developer, I was working on reports which involved lovingly handcrafted (and oftentimes quite complicated) SQL queries.

read more (365 words)

Posted by under post at / software development sql / Syndicated: / 0 / 365 words
Despite our best efforts as software developers, it can still happen: production goes down. Or some sort of bug introduces catastrophic data error. Hopefully you have a support/DevOps team to handle the response. If not, the dev team themselves have to step in. This usual means a mad rush to figure out what happened and how to fix it, sometimes during off hours and maybe even into the early morning, all while facing pressure from clients and higher-ups.

read more (441 words)

Posted by under post at / software development / Syndicated: / 1 1 / 441 words
I’ve been using Python for well over 10 years, and I still don’t have an intuitive mastery of one of its keywords: yield. Everytime I see it in someone’s code I need to stop and mentally remind myself what it does. I figured I’d write a devnote to help improve my recall. Typically, yield is used in a function with a loop, like so: def some_func(lim): for i in range(0, lim): yield i yield means the function returns a “generator” that can be used as an iterable in a loop:

read more (172 words)

Posted by under post at / software development devnotes / Syndicated: / 1 0 / 172 words
A while back I found myself having to figure out how to compile/build/run a mobile application. The developers previously assigned to the project were no longer available to consult with, but they did leave behind some documentation. However, their documentation quality left a lot to be desired. The instructions they left basically amounted to: npm install ionic serve ionic codrova run android/ios Okay, first sign of trouble is that their instructions were basically commands that anyone who knew the app used Ionic would be able to Google.

read more (725 words)

Posted by under post at / software development / Syndicated: / 0 / 725 words

Aug 2019

Back when I was still learning Python in 2008, one of the first “fun” scripts I wrote was a text generator using Markov chains. I’d run it against all the chat logs I had with people at work and serve the results from a webserver on my computer. THe results were often amusing and sometimes hilarious. Since I’ve been going through my old scripts lately, I thought I’d update that script to Python 3 (read: add parentheses around print params and use pathlib) and run it against all the posts on this here site.

read more (162 words)

Posted by under post at / software development python / Syndicated: / 0 / 162 words
Ever since I started learning Python back in 2008ish, I’ve been using it as my primary scripting language for various tasks such as processing log files, organizing my own file system, processing stuff on this blog, and so on. A lot of it is basically moving files around. In the days of Python 2, that involved a lot of imports of different libraries like os, shutil and glob. It can become a bit messy with so many imports, and I often can’t remember which import I need for a particular case and end up having to search for the documentation (or stackoverflow, let’s not kid ourselves here).

read more (316 words)

Posted by under post at / software development devnotes python / Syndicated: / 0 / 316 words
Big news in online repositories this week is that Bitbucket is sunsetting support for Mercurial! This might be the death knell for Mercurial, although Git was already the super popular choice before. Back when I started using online source control for my personal coding projects I started out with Bitbucket over Github because they offered unlimited private repos and Mercurial (which I had already tried out before at work, so at first I preferred it over git).

read more (279 words)

Posted by under post at / software development tech life devnotes / Syndicated: / 0 / 279 words
I recently found myself doing a really small project as sort of a proof of concept/demo for a potential client. It often seems that it might be a waste of time to do something like this since you don’t know if the project will actually push through or maybe the client will want something else. To kind of hedge my bets a bit, I decided to take the opportunity to try out some new technologies so that no matter what I at least learned something from all of this.

read more (750 words)

Posted by under post at / devnotes software development / Syndicated: / 1 0 / 750 words
I decided to start doing small “devnotes” on developer stuff I’m doing so I can refer to them later (and also because I feel like I could use more technical content on this blog) Today is about PostgreSQL. I haven’t used it much beyond standard ANSI sql stuff. You won’t always have a graphical interface to access your database, sometimes you need to ssh to prod and query the database from the shell.

read more (317 words)

Posted by under post at / devnotes software development / Syndicated: / 0 / 317 words

Jul 2019

I have a small mobile app that I wrote using React Native (henceforth RN) back in 2017, currently deployed on the Google Play Store and Apple App Store. Shortly before my US trip, I got an email from Google telling me about a required action: By August 1, 2019, all apps that use native code must provide a 64-bit version in addition to the 32-bit version in order to publish an update.

read more (1080 words)

Posted by under post at / software development react native mobile / Syndicated: / 0 / 1080 words
The topic of the mythical “10x programmer” has been the topic of discussion recently on tech twitter, due to a thread listing out the supposed signs of being such a mythical beast. 10x engineers Founders if you ever come across this rare breed of engineers, grab them. If you have a 10x engineer as part of your first few engineers, you increase the odds of your startup success significantly. OK, here is a tough question.

read more (1948 words)

Posted by under post at / software development / Syndicated: / 0 / 1948 words

Apr 2019

Success is not final, failure is not fatal: it is the courage to continue that counts. – Winston Churchill I already typed the above quote into the post, then realized I had already used it before. Whatever, just goes to show, I’m no stranger to failure. I was reminded of this quote because recently I prepared a demo for a project that didn’t push through. At first I was annoyed at the wasted effort, but I realized that I had wisely taken the demo project as an opportunity to learn/sharpen some skills.

read more (155 words)

Posted by under post at / quotes self-improvement software development / Syndicated: / 0 / 155 words

Mar 2019

I wish I had a more concise way to describe it, but I really don’t. Some time ago this guy I follow on Twitter, visakanv wanted to know how to do a certain search: he wanted to know who a given famous person follows on Twitter, and among those, finds the one who follow him (visakanv), so he could network through them. I might not be explaining the concept too well, here’s the thread.

read more (373 words)

Posted by under post at / Software Development / Syndicated: / 0 / 373 words
I mentioned before that as an engineer, I’m not fond of marketing. Image credit: Dilbert.com (Disclaimer: Liking the Dilbert comics is not an endorsement of Scott Adams’ politics) It’s not that I can’t be good at salesmanship either. I have a good grasp of communication skills and think I have a decent chance of writing good copy. My main issue is that I’ve been exposed many times to sales/marketing practices that just seem dishonest downright or scummy.

read more (312 words)

Posted by under post at / Software Development / 0 / 312 words
If I could give some advice to someone starting out in their software development career, it would be this: Don’t stay in the same place too long. The first company I worked at, I stayed with them for thirteen years, which I now feel was way too long. I have to admit, the work was hard and challenging, but I was young and had a lot of energy and was willing to work the long hours.

read more (480 words)

Posted by under post at / Software Development / Syndicated: / 0 / 480 words
Another repost from my Quora answers, this time some info for anyone looking to move into programming. What are the pros and cons of making your career in programming? Pros: It is a very rewarding career financially. Software development often ranks in the top 10 highest-earning careers in most countries There is a lot of scope - you could be developing web applications, mobile applications, embedded applications, client-side, server-side, data analysis, artificial intelligence, games, etc It is very difficult to be bored.

read more (459 words)

Posted by under post at / quora Software Development / Syndicated: / 0 / 459 words
Given my recent misgivings about Quora, I thought it might be a good idea to cross-post some of my answers from there into this blog, with some edits even. So here’s the first one! (stuff in italics were added during the cross-post) How can you read and study a large software project source code? Attacking a large, existing codebase that you are unfamiliar with can be a daunting endeavor.

read more (393 words)

Posted by under post at / quora Software Development / Syndicated: / 0 / 393 words

Feb 2019

I had some free time the other day so I randomly decided to post in the PH subreddit’s regular afternoon random discussion thread, asking for questions about software development. I ended up typing some longish answers, I thought I’d copy them over to the blog in case anyone was interested. TBH I meant more like StackOverflow type questions with specific technical problems, but I ended up answering mostly career-related questions, which is fine, but disclaimer: I don’t claim to be an expert, these are just my opinions on things.

read more (1124 words)

Posted by under post at / reddit Software Development Philippines favorites / Syndicated: / 1 / 1124 words
Last October I participated in #Hacktoberfest, sponsored by DigitalOcean and Github. It’s a “celebration” to promote open source activity, and basically you just need to submit 5 pull requests to any github repository, and they give away swag to anyone who completes the activity. Microsoft held a [counterpart celebration] where they only require you to submit 1 pull request to any Microsoft repository. I’ve always wanted to start participating in Open Source, but it’s a bit difficult to find a good place to contribute (other than logging issues of course).

read more (386 words)

Posted by under post at / swag hacktoberfest open source Software Development / Syndicated: / 0 / 386 words
A while back I started a Twitter trivia bot as a weekend project. That bot is still up and running on Twitter, you can check it out there! But today, I thought I’d write about the answer-checking mechanism used by the bot. It was a bit interesting to me because it was the first nontrivial use I had for Django’s unit testing framework. I’m not too keen on unit testing web functionality (something I still have to learn), but this seemed an appropriate first use of a unit test framework for several reasons:

read more (579 words)

Jan 2019

I’ve been working with Javascript for more than a decade. Last week while helping another developer debug a problem, I had to Google how to check if an element exists in a Javascript array, something superbasic, that one would expect most newbies to know. I’m sure I Google some superbasic thing at least once a week. It’s not embarassing or anything, it’s a common occurrence. I’m surely not alone. Just last night a tweet about this crossed my TL:

read more (340 words)

Posted by under post at / Javascript Software Development / Syndicated: / 0 / 340 words
Someone responded to my post on things to learn in 2019 by asking how one finds the inspiration to learn all of the things. Well, my first answer was that those are just things I find interesting and may look into, but that’s not really an answer for the inspiration part. Software development is a very wide field, one where the amount of things you can learn increases daily, so it’s almost impossible to keep up with everything.

read more (435 words)

Posted by under post at / Software Development / Syndicated: / 0 / 435 words
A while back we were tasked with helping a client’s internal dev team to migrate their repositories from Subversion to Git. The distributed VCS seemed ideal for their situation - they had a very small in-house dev team managing contributions from external subcontractors. The main rationale was that their process of merging contributions from the external developers was extremely complicated and often resulted in conflicts that were challenging to merge. Before this, I hadn’t actually used Git too deeply myself (aside from cloning stuff from Github), and especially not in a team setting, so the training one of our other engineers gave them was a good opportunity for me to become familiar with Git as well.

read more (702 words)

Posted by under post at / Software Development / Syndicated: / 0 / 702 words
One of the things about self-identifying as a “Full Stack Developer” or “Solution Architect” is that there’s no shortage of things to learn, and oftentimes it’s good for your career-wise to at least have some passing knowledge of a bunch of technologies. It helps that I really like the field as well. I try to make sure I study or learn at least one new programming language or framework every year (though I am willing to stretch that definition as needed).

read more (691 words)

Posted by under post at / Software Development / Syndicated: / 0 / 691 words
Although I still primarily identify as a “Full Stack Developer”, during the past few years I’ve also found myself in a role called “Solution Architect”. The thing about being a solution architect is that there isn’t really a clear definition of the role, what it involves, or the scope of responsibility. I suppose it depends largely on the organization and the project. The role mostly involves making techical decisions on a larger scale, like project-wide or organization-wide, rather than on the micro day-to-day technical decisions involved in a typical software development involves.

read more (380 words)

Posted by under post at / Software Development / Syndicated: / 0 / 380 words
This year I had the dubious privilege of having to work with a C++ project again. Although my college education was in C, that was a completely different animal. I did self-study C++ for a bit back even before I was working, mostly because I was interested in game development even back then. I remember trying some OpenGL and/or DirectX stuff back with good old Borland Turbo-C++ during the DOS days and using the Dev-C++ IDE when I shifted to Windows.

read more (956 words)

Posted by under post at / c++ Software Development / Syndicated: / 0 / 956 words

Dec 2018

For any nontrivial software project of at least moderate team size, there can be a significant cost to onboarding a new team member, especially at later stages when you are rushing to meet deadlines. The most signifiant cost is of course the communication overhead as described in the Mythical Man Month. Fun story, the CEO of a company once told me they would add more developers to a delayed project to meet the deadline and when I pointed out the increased overhead he said to me that it wasn’t a problem because they would just assign modules to those devs that have minimal dependencies so they don’t have to communicate so much.

read more (790 words)

Posted by under post at / devto Software Development / Syndicated: / 2 / 790 words

Nov 2018

Text editors (and by extension IDEs) are a programmer’s best friend. I thought I’d look back at a number of text editors I’ve used over the years. (I grew up with Windows, so I won’t list vim/emacs/nano here, even though I’m at least a bit proficient with vim by now. That is, I know how to exit vim.) Notepad – of course, the default editor in Windows. The one we turn to when all else fails.

read more (666 words)

Posted by under post at / Tech Life Software Development / Syndicated: / 3 / 666 words
According to Malcolm Gladwell’s book Outliers, you need 10,000 hours of continuous sustained practice to become an expert. There are 168 hours in a week. If you never sleep and you eat as you practice, you can become an expert in 60 weeks. (Around 14 months) If you sleep 8 hours a day, you only have 112 hours in a week. If you eat as you practice, you can become an expert in 90 weeks.

read more (260 words)

Posted by under post at / Self-Improvement Software Development / Syndicated: / 0 / 260 words
SCM (Software Configuration Management) doesn’t just refer to version control for the software you’re building. It also means controlling the versions of software you depend on. This includes operating system and programming runtimes. Sometimes even minor version differences can cause issues in running your software. I have two example stories to share: One of our clients asked us for help with an upgrade their production servers from CentOS 6.4 to 6.

read more (320 words)

Posted by under post at / Software Development / Syndicated: / 0 / 320 words

Oct 2018

Systemic change is difficult. I’m talking about software projects/systems, but there are a lot of parallels with societal systems too, like governments or states. I’ve been in large projects with hundreds of thousands of LOC where a lot of the code was painful to read and full of code smells and so on. It happens over time as projects get maintained by different developers and teams or different enhancements or changes are made.

read more (551 words)

Posted by under post at / Politics Philippines Opinions Software Development / Syndicated: / 0 / 551 words
Ten years ago this month, I started studying Django by trying to build my own blog application. I found the code lying around while I was going through some backups lately. It’s way out of date, it uses an early version of django. I thought of bringing it up to speed, but that didn’t seem practical. Instead, for archival purposes, I cleaned it up a bit and uploaded the code to a github repo.

read more (176 words)

Posted by under post at / projects python royondjango Software Development / Syndicated: / 0 / 176 words
Malcolm Gladwell, in an article from 1996 discussing the Challenger disaster, tells us: This kind of disaster is what the Yale University sociologist Charles Perrow has famously called a “normal accident.” By “normal” Perrow does not mean that it is frequent; he means that it is the kind of accident one can expect in the normal functioning of a technologically complex operation. Modern systems, Perrow argues, are made up of thousands of parts, all of which interrelate in ways that are impossible to anticipate.

read more (523 words)

Posted by under post at / risk-management Software Development / Syndicated: / 0 / 523 words
Rockstar was in the gaming news recently because they mentioned that some of them had worked 100-hour weeks on their massive sequel to Red Dead Redemption coming out soon (no idea if I’ll play this). The idea of 100 hour weeks seemed insane to me, and it got me thinking: I’ve done some serious overtime before, have I ever gotten close to that amount of work in a week? Luckily, I didn’t have to speculate too much, because I had data (I love data).

read more (294 words)

Posted by under post at / Gaming Software Development / Syndicated: / 0 / 294 words
Mentoring is one of those tasks that’s to be expected of anyone in a senior software development role. This usually involves reviewing other people’s code, helping them with tough technical issues, and even giving career advice. I’m not sure how good I am when it comes to mentoring other software developers. When I first became technical lead on projects, I got some evaluations from junior developers saying I can be “intimidating”.

read more (550 words)

Posted by under post at / Software Development / Syndicated: / 0 / 550 words

Apr 2018

As a programmer, I’ve always been a big fan of StackOverflow. I asked my first question there and also wrote my first answer in September 2008, which was the month the site launched, so I was pretty much there from the beginning. The site was a huge boon to programmers when it first came out, because the internet as a venue for asking questions and answers back then was a horrible fragmented landscape of small forums and mailing lists and sites like Experts Exchange, all of which were terribly designed.

read more (969 words)

Posted by under post at / Software Development / Syndicated: / 0 / 969 words

Mar 2018

While browsing through my old blog posts, I found one about my setup from 2010. I figured it was a good time to do an update. I like doing posts like these because it provides an easy reference for me to look back and see what I was working with at a certain point in time. What Hardware Do I Use? Desktop. I bought a new desktop rig back in late 2015, here are the specs:

read more (1451 words)

Posted by under post at / Gaming Software Development Tech Life / Syndicated: / 2 / 1451 words

Feb 2018

According to the documentation here: https://djangobook.com/syndication-feed-framework/

If link doesn’t return the domain, the syndication framework will insert the domain of the current site, according to your SITE_ID setting

However, I’m trying to generate a feed of magnet: links. The framework doesn’t recognize this and attempts to append the SITE_ID, such that the links end up like this (on localhost):

<link>http://localhost:8000magnet:?xt=...</link>

Is there a way to bypass this?

Dec 2017

I have a query using MarkLogic node.js that basically boils down to something like this:

db.documents.query(qb.where(qb.collection('test'))).stream()
.on('data', function(row) {
    console.log("Stream on data");
})
.on('end', function() {
    console.log("Stream on end");
})
.on('error', function(error) {
    console.log(error);
})
;

Now, for a certain collection we have in our database, the ‘end’ function doesn’t fire, i.e. I never see “Stream on end” appear in the log. There’s no error or anything, processing just stops. It’s only for this particular collection, other collections seem fine.

If I query documents in that collection directly using other methods such as qb.value() without using qb.collection(), the end event fires correctly. But once I add qb.collection() into the mix (using qb.and), the end event doesn’t fire.

I’m unsure how to debug this, as this is my first time trying to use streams in the nodejs client library. Any advice as to what I can check?

Thanks!

I was thinking about my typical approach to coding. When writing a new feature, I tend to implement in the direction of where the data flows, starting from the user interface then to the backend and back to the frontend and wherever else that goes. I will build incrementally, using debugging tools or console printouts to ensure that each step is working correctly. As an example, here’s how I did a recent web-based function:

read more (916 words)

Posted by under post at / Software Development / Syndicated: / 0 / 916 words

How to specify that a column in the schema should be nullable?

I tried adding a nullable attribute:

var myFirstTDE = xdmp.toJSON(
  {
    "template": {
      "context": "/match",
      "collections": ["source1"],
      "rows": [
        {
          "schemaName": "soccer",
          "viewName": "matches",
          "columns": [
            {
              "name": "id",
              "scalarType": "long",
              "val": "id",
              "nullable": 0
            },
            {
              "name": "document",
              "scalarType": "string",
              "val": "docUri"
            },
            {
              "name": "date",
              "scalarType": "date",
              "val": "match-date"
            },
            {
              "name": "league",
              "scalarType": "string",
              "val": "league"
            }
          ]
        }
      ]
    }
  }
);

tde.validate( 
  [myFirstTDE]
);

But this gave me a template error:

"message": "TDE-INVALIDTEMPLATENODE: Invalid extraction template node: fn:doc('')/template/array-node('rows')/object-node()/array-node('columns')/object-node()[1]/number-node('nullable')"

For a template defined using XQuery, adding nullable to the column works:

<column>
  <name>ISSN</name>
  <scalar-type>string</scalar-type>
  <val>Journal/ISSN</val>
  <nullable>true</nullable>
</column>

How to do the same thing using JS/Json?

Nov 2017

This is a follow-up to my question here: https://stackoverflow.com/questions/47449002/marklogic-template-driven-extraction-and-triples-dealing-with-array-nodes/47459250#47459250

So let’s say I have a number of documents structured like this:

declareUpdate();

xdmp.documentInsert(
       '/test/tde.json',
       {
         content: {
           name:'Joe Parent',
           children: [
             {
               name: 'Bob Child'
             },
             {
               name: 'Sue Child'
             },
             {
               name: 'Guy Child'
             }
           ]
         }
       },
       {permissions : xdmp.defaultPermissions(),
        collections : ['test']})

I want to define a template that would extract triples from these documents defining sibling relationships between the children. For the above example, I would want to extract the following triples (the relationship is two-way):

Bob Child sibling-of Sue Child
Bob Child sibling-of Guy Child
Sue Child sibling-of Bob Child
Sue Child sibling-of Guy Child
Guy Child sibling-of Bob Child
Guy Child sibling-of Sue Child

How can i set up my template to accomplish this?

Thanks!

I’ve been studying the examples here: https://docs.marklogic.com/guide/semantics/tde#id_25531

I have a set of documents that are structured with a parent name and an array of children nodes with their own names. I want to create a template that generates triples of the form “name1 is-a-parent-of name2”. Here’s a test I tried, with a sample of the document structure:

declareUpdate();

xdmp.documentInsert(
       '/test/tde.json',
       {
         content: {
           name:'Joe Parent',
           children: [
             {
               name: 'Bob Child'
             },
             {
               name: 'Sue Child'
             }
           ]
         }
       },
       {permissions : xdmp.defaultPermissions(),
        collections : ['test']})

cts.doc('/test/tde.json')

var tde = require("/MarkLogic/tde.xqy");
// Load the user template for user profile rows
var template = xdmp.toJSON(
{
  "template":{
    "context":"content",
    "collections": [
      "test"
    ],
      "triples":[
      {
        "subject": {
          "val": "xs:string(name)"
        },
        "predicate": {
          "val": "sem:iri('is-parent-of')"
        },
        "object": {
          "val": "xs:string(children/name)"
        }     
      }
    ]   
  }
}
);
//tde.validate([template]),
tde.templateInsert("/templates/test.tde", template);
tde.nodeDataExtract( 
  [cts.doc( '/test/tde.json' )]
)

However, the above throws an Exception:

[javascript] TDE-EVALFAILED: tde.nodeDataExtract([cts.doc(“/test/tde.json”)]) – Eval for Object=‘xs:string(children/name)’ returns TDE-BADVALEXPRESSION: Invalid val expression: XDMP-CAST: (err:FORG0001) Invalid cast: (fn:doc(“/test/tde.json”)/content/array-node(“children”)/object-node()[1]/text(“name”), fn:doc(“/test/tde.json”)/content/array-node(“children”)/object-node()[2]/text(“name”)) cast as xs:string?

What is the proper syntax for extracting array nodes into a triple?

2nd somewhat related question: say I also wanted to have triples of the form “child1 is-sibling-of child2”. For the example above it would be “Bob Child is-sibling-of Sue Child”. What would be the proper syntax for this? I’m not even sure how to begin with this one.

Is TDE even the way to go here? Or is it better to do this programmatically? i.e. on document ingestion, generate those triples inside the document directly?

(If it’s relevant, the ML version being used is 9.)

I’ve been testing migrating one of our systems to Marklogic 9 and using the Optics API.

One of our functions involves grouping claims by member_id, member_name and getting the sums and counts, so I did something like this:

var results = op.fromView('test', 'claims')
  .groupBy(['member_id', 'member_name'], [
         op.count('num_claims', 'claim_no'),
         op.sum('total_amount', 'claim_amount')
         ])
  .orderBy(op.desc('total_amount'))
  .limit(200)
  .result()
  .toArray();

Above works fine. The results are of the form

[
  { 
    member_id: 1, 
    member_name: 'Bob', 
    num_claims: 10, 
    total_amount: 500
  }, 
  ...
]

However, we also have a field “company”, where each claim is filed under a different company. Basically the relevant view columns are claim_no, member_id, member_name, company, claim_amount

I would like to be able to show a column that list the different companies for which the member_id/member_name has filed claims, and how many claims for each company.

i.e. I want my results to be something like:

[
  { 
    member_id: 1, 
    member_name: 'Bob', 
    num_claims: 10, 
    total_amount: 500,
    companies: [
      {
        company: 'Ajax Co',
        num_claims: 8
      },
      {
        company: 'Side Gig',
        num_claims: 2
      }
    ]
  }, 
  ...
]

I tried something like this:

results = results.map((member, index, array) => {
  var companies = op.fromView('test', 'claims')
    .where(op.eq(op.col('member_id'), member.member_id))
    .groupBy('company', [
      op.count('num_claims', 'claim_no')      
    ])
    .result()
    .toArray();
  member.companies = companies;
  return member;
});

And the output seems correct, but it also executes quite slowly - almost a minute (total number of claim documents is around 120k)

In our previous ML8 implementation, we were pre-generating summary documents for each member - so retrieval was reasonably fast with the downside that whenever we got a bunch of new data, all of the summary documents had to be re-generated. I was hoping that ML9’s optic API would make it easier to do the retrieval/grouping/aggregates on the fly so we wouldn’t have to do that.

In theory, I could just add company to the groupBy fields, then merge the rows in the result query as needed. But the problem with that approach is that I can’t guarantee I’ll get the top 200 by total amount (as was my original query)

So, the question is: Is there a better way of doing this with a reasonable execution time? Or should I just stick to pre-generating the summary documents?

I have a Marklogic 9 project that I’m configuring with Roxy. I’ve been following these examples: https://github.com/marklogic-community/roxy/wiki/Adding-Custom-Build-Steps

Basically, I have a server-side JS function that I want to call after deploy content. I have something like this:

# then you would define your new method

  def deploy_content
    # you can optionally call the original
    original_deploy_content

    # do your stuff here
    execute_query(%Q{
      xquery version "1.0-ml";
      xdmp:javascript-eval('var process = require("/ingestion/process.sjs"); process.postDeployContent();')
    },
    :db_name => @properties["ml.app-name"] + "-content")

  end

The xquery being called here evaluates fine when executed via query console. But when I call ml local deploy content, I get the following error:

ERROR: 500 "Internal Server Error"
ERROR: <html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>500 Internal Server Error</title>
    <meta name="robots" content="noindex,nofollow"/>
    <link rel="stylesheet" href="/error.css"/>
  </head>
  <body>
    <span class="error">
      <h1>500 Internal Server Error</h1>
      <dl>
        <dt>XDMP-MODNOTFOUND: var process = require("/ingestion/process.sjs"); process.postDeployContent(); -- Module /ingestion/process.sjs not found</dt>
        <dd></dd>
        <dt>in [anonymous], at 1:14 [javascript]</dt>
        <dd></dd>
        <dt>at 3:6,
in xdmp:eval("var process = require(&amp;quot;/ingestion/process.sjs&amp;quot;); proce...") [javascript]</dt>
        <dd></dd>
        <dt>in /eval, at 3:6 [1.0-ml]</dt>
        <dd></dd>
      </dl>
    </span>
  </body>
</html>

Why is the module not found when running via xquery from app_specific.rb?

Or… is there a better way to call a JS module function from here. Sorry, I’m not too familiar with the xquery side, so I just called a JS function instead.

Basically the title. The client is complaining that when he zooms in, the text labels for the nodes are quite large. Is there a way to keep the node labels at a fixed font size even when zooming in or out?

From the nodes documentation (http://visjs.org/docs/network/nodes.html), there’s a scaling.label option, but it doesn’t seem to work. I think this is only relevant if I’m using values to scale the nodes.

Oct 2017

I’m trying out the Roxy deployer. The Roxy app was created using the default app-type. I setup a new ML 9 database, and I ran “ml local bootstrap” using the default ports (8040 and 8041)

Then I setup a node application. I tried the following (sample code from https://docs.marklogic.com/jsdoc/index.html)

var marklogic = require('marklogic');
var conn = {
    host: '192.168.33.10',  
    port: 8040,
    user: 'admin',
    password: 'admin',
    authType: 'DIGEST'
}

var db = marklogic.createDatabaseClient(conn);

db.createCollection(
  '/books',
  {author: 'Beryl Markham'},
  {author: 'WG Sebald'}
  )
.result(function(response) {
    console.log(JSON.stringify(response, null, 2));
  }, function (error) {
    console.log(JSON.stringify(error, null, 2));
  });

Running the script gave me an error like:

$ node test.js
{
  "message": "write document list: cannot process response with 500 status",
  "statusCode": 500,
  "body": "<error:error xsi:schemaLocation=\"http://marklogic.com/xdmp/error error.xsd\" xmlns:error=\"http://marklogic.com/xdmp/error\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n  <error:code>XDMP-IMPMODNS</error:code>\n  <error:name>err:XQST0059</error:name>\n  <error:xquery-version>1.0-ml</error:xquery-version>\n  <error:message>Import module namespace mismatch</error:message>\n  <error:format-string>XDMP-IMPMODNS: (err:XQST0059) Import module namespace http://marklogic.com/rest-api/endpoints/config does not match target namespace http://marklogic.com/rest-api/endpoints/config_DELETE_IF_UNUSED of imported module /MarkLogic/rest-api/endpoints/config.xqy</error:format-string>\n  <error:retryable>false</error:retryable>\n  <error:expr/>\n  <error:data>\n    <error:datum>http://marklogic.com/rest-api/endpoints/config</error:datum>\n    <error:datum>http://marklogic.com/rest-api/endpoints/config_DELETE_IF_UNUSED</error:datum>\n    <error:datum>/MarkLogic/rest-api/endpoints/config.xqy</error:datum>\n  </error:data>\n  <error:stack>\n    <error:frame>\n      <error:uri>/roxy/lib/rewriter-lib.xqy</error:uri>\n      <error:line>5</error:line>\n      <error:column>0</error:column>\n      <error:xquery-version>1.0-ml</error:xquery-version>\n    </error:frame>\n  </error:stack>\n</error:error>\n"
}

If I change the port to 8000 (the default appserver that inserts into Documents), the node function executes correctly as expected. I’m not sure if I need to configure anything else with the Roxy-created appserver so that it works with the node.js application.

I’m not sure where the “DELETE_IF_UNUSED” part in the error message is coming from either. There doesn’t seem to be any such text in the configuration files generated by Roxy.

Edit: When accessing 192.168.33.10:8040 via the browser, I get a an xml with a similar error:

<error:error xsi:schemaLocation="http://marklogic.com/xdmp/error error.xsd" xmlns:error="http://marklogic.com/xdmp/error" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <error:code>XDMP-IMPMODNS</error:code>
  <error:name>err:XQST0059</error:name>
  <error:xquery-version>1.0-ml</error:xquery-version>
  <error:message>Import module namespace mismatch</error:message>
  <error:format-string>XDMP-IMPMODNS: (err:XQST0059) Import module namespace http://marklogic.com/rest-api/endpoints/config does not match target namespace http://marklogic.com/rest-api/endpoints/config_DELETE_IF_UNUSED of imported module /MarkLogic/rest-api/endpoints/config.xqy</error:format-string>
  <error:retryable>false</error:retryable>
  <error:expr/>
  <error:data>
    <error:datum>http://marklogic.com/rest-api/endpoints/config</error:datum>
    <error:datum>http://marklogic.com/rest-api/endpoints/config_DELETE_IF_UNUSED</error:datum>
    <error:datum>/MarkLogic/rest-api/endpoints/config.xqy</error:datum>
  </error:data>
  <error:stack>
    <error:frame>
      <error:uri>/roxy/lib/rewriter-lib.xqy</error:uri>
      <error:line>5</error:line>
      <error:column>0</error:column>
      <error:xquery-version>1.0-ml</error:xquery-version>
    </error:frame>
  </error:stack>
</error:error>

If it matters, MarkLogic version is 9.0-3.1. It’s a fresh install too.

Any advice?

I’m trying to migrate one of my dev envts from ML8 to ML9. I have an import script that successfully works on the ML8 version, but there’s an error when I try running it against the ML9 database. The ML9 version is 9.0.3.1. The MLCP version is 9.0.3

My MLCP options file is as follows:

import
-host 
192.168.33.10
-port 
8041
-username 
admin
-password
admin
-input_file_path 
d:\maroon\data\mbastest.csv 
-mode 
local 
-input_file_type 
delimited_text 
-uri_id 
ClientId 
-output_uri_prefix
/test/records/
-output_uri_suffix 
.json 
-document_type 
json 
-transform_module 
/ingestion/transform.js
-transform_function 
testTransform
-transform_param
test
-content_encoding 
windows-1252 
-thread_count
1

Here’s the output of a test run with only 2 records in the test CSV file:

17/10/30 14:07:33 INFO contentpump.LocalJobRunner: Content type: JSON
17/10/30 14:07:33 INFO contentpump.ContentPump: Job name: local_455168344_1
17/10/30 14:07:33 INFO contentpump.FileAndDirectoryInputFormat: Total input paths to process : 1
17/10/30 14:07:38 WARN contentpump.TransformWriter: Failed document /test/records/31.json
17/10/30 14:07:38 WARN contentpump.TransformWriter: <error:format-string xmlns:error="http://marklogic.com/xdmp/error" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected QName_, expecting $end or SemiColon_</error:format-string>
17/10/30 14:07:38 WARN contentpump.TransformWriter: Failed document /test/records/32.json
17/10/30 14:07:38 WARN contentpump.TransformWriter: <error:format-string xmlns:error="http://marklogic.com/xdmp/error" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected QName_, expecting $end or SemiColon_</error:format-string>
17/10/30 14:07:38 INFO contentpump.LocalJobRunner:  completed 100%
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: com.marklogic.mapreduce.MarkLogicCounter:
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: INPUT_RECORDS: 2
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS: 2
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_COMMITTED: 0
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_FAILED: 2
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: Total execution time: 5 sec

If I remove the transform params, the import works fine.

I thought it might be a parsing issue with my transform module itself, so I tried replacing it with the following example from the documentation:

// Add a property named "NEWPROP" to any JSON input document.
// Otherwise, input passes through unchanged.

function addProp(content, context)
{
  const propVal = (context.transform_param == undefined)
                 ? "UNDEFINED" : context.transform_param;

  if (xdmp.nodeKind(content.value) == 'document' &&
      content.value.documentFormat == 'JSON') {
    // Convert input to mutable object and add new property
    const newDoc = content.value.toObject();
    newDoc.NEWPROP = propVal;

    // Convert result back into a document
    content.value = xdmp.unquote(xdmp.quote(newDoc));
  }
  return content;
};

exports.addProp = addProp;

(Of course I changed the params in the MLCP options file accordingly)

The issue still persists even with just this test function.

Any advice?

I’ve been testing out the WebView component, but I can’t seem to get it to render things.

Sample here: https://snack.expo.io/r1oje4C3-

export default class App extends Component {
  render() {
    return (
      <View style={styles.container}>
        <Text style={styles.paragraph}>
          Change code in the editor and watch it change on your phone!
          Save to get a shareable url. You get a new url each time you save.
        </Text>
        <WebView source={{html: '<p>Here I am</p>'}} />        
        <WebView source={{ uri: 'http://www.google.com'}} />
      </View>
    );
  }
}

When running the above example in Expo, neither of the two WebView components seem to render. What am I doing wrong?

Roy Tang is a:

roytang.net is a personal site, an E/N site, and kind of a commonplace book; I post about a random assortment of topics that interest me including software development, Magic the Gathering, pop culture, gaming, and tech life. This site is perpetually under renovation.