Roy Tang

According to the documentation here: https://djangobook.com/syndication-feed-framework/

If link doesn’t return the domain, the syndication framework will insert the domain of the current site, according to your SITE_ID setting

However, I’m trying to generate a feed of magnet: links. The framework doesn’t recognize this and attempts to append the SITE_ID, such that the links end up like this (on localhost):

<link>http://localhost:8000magnet:?xt=...</link>

Is there a way to bypass this?

I have a query using MarkLogic node.js that basically boils down to something like this:

db.documents.query(qb.where(qb.collection('test'))).stream()
.on('data', function(row) {
    console.log("Stream on data");
})
.on('end', function() {
    console.log("Stream on end");
})
.on('error', function(error) {
    console.log(error);
})
;

Now, for a certain collection we have in our database, the ‘end’ function doesn’t fire, i.e. I never see “Stream on end” appear in the log. There’s no error or anything, processing just stops. It’s only for this particular collection, other collections seem fine.

If I query documents in that collection directly using other methods such as qb.value() without using qb.collection(), the end event fires correctly. But once I add qb.collection() into the mix (using qb.and), the end event doesn’t fire.

I’m unsure how to debug this, as this is my first time trying to use streams in the nodejs client library. Any advice as to what I can check?

Thanks!

How to specify that a column in the schema should be nullable?

I tried adding a nullable attribute:

var myFirstTDE = xdmp.toJSON(
  {
    "template": {
      "context": "/match",
      "collections": ["source1"],
      "rows": [
        {
          "schemaName": "soccer",
          "viewName": "matches",
          "columns": [
            {
              "name": "id",
              "scalarType": "long",
              "val": "id",
              "nullable": 0
            },
            {
              "name": "document",
              "scalarType": "string",
              "val": "docUri"
            },
            {
              "name": "date",
              "scalarType": "date",
              "val": "match-date"
            },
            {
              "name": "league",
              "scalarType": "string",
              "val": "league"
            }
          ]
        }
      ]
    }
  }
);

tde.validate( 
  [myFirstTDE]
);

But this gave me a template error:

"message": "TDE-INVALIDTEMPLATENODE: Invalid extraction template node: fn:doc('')/template/array-node('rows')/object-node()/array-node('columns')/object-node()[1]/number-node('nullable')"

For a template defined using XQuery, adding nullable to the column works:

<column>
  <name>ISSN</name>
  <scalar-type>string</scalar-type>
  <val>Journal/ISSN</val>
  <nullable>true</nullable>
</column>

How to do the same thing using JS/Json?

This is a follow-up to my question here: https://stackoverflow.com/questions/47449002/marklogic-template-driven-extraction-and-triples-dealing-with-array-nodes/47459250#47459250

So let’s say I have a number of documents structured like this:

declareUpdate();

xdmp.documentInsert(
       '/test/tde.json',
       {
         content: {
           name:'Joe Parent',
           children: [
             {
               name: 'Bob Child'
             },
             {
               name: 'Sue Child'
             },
             {
               name: 'Guy Child'
             }
           ]
         }
       },
       {permissions : xdmp.defaultPermissions(),
        collections : ['test']})

I want to define a template that would extract triples from these documents defining sibling relationships between the children. For the above example, I would want to extract the following triples (the relationship is two-way):

Bob Child sibling-of Sue Child
Bob Child sibling-of Guy Child
Sue Child sibling-of Bob Child
Sue Child sibling-of Guy Child
Guy Child sibling-of Bob Child
Guy Child sibling-of Sue Child

How can i set up my template to accomplish this?

Thanks!

I’ve been studying the examples here: https://docs.marklogic.com/guide/semantics/tde#id_25531

I have a set of documents that are structured with a parent name and an array of children nodes with their own names. I want to create a template that generates triples of the form “name1 is-a-parent-of name2”. Here’s a test I tried, with a sample of the document structure:

declareUpdate();

xdmp.documentInsert(
       '/test/tde.json',
       {
         content: {
           name:'Joe Parent',
           children: [
             {
               name: 'Bob Child'
             },
             {
               name: 'Sue Child'
             }
           ]
         }
       },
       {permissions : xdmp.defaultPermissions(),
        collections : ['test']})

cts.doc('/test/tde.json')

var tde = require("/MarkLogic/tde.xqy");
// Load the user template for user profile rows
var template = xdmp.toJSON(
{
  "template":{
    "context":"content",
    "collections": [
      "test"
    ],
      "triples":[
      {
        "subject": {
          "val": "xs:string(name)"
        },
        "predicate": {
          "val": "sem:iri('is-parent-of')"
        },
        "object": {
          "val": "xs:string(children/name)"
        }     
      }
    ]   
  }
}
);
//tde.validate([template]),
tde.templateInsert("/templates/test.tde", template);
tde.nodeDataExtract( 
  [cts.doc( '/test/tde.json' )]
)

However, the above throws an Exception:

[javascript] TDE-EVALFAILED: tde.nodeDataExtract([cts.doc("/test/tde.json”)]) – Eval for Object='xs:string(children/name)’ returns TDE-BADVALEXPRESSION: Invalid val expression: XDMP-CAST: (err:FORG0001) Invalid cast: (fn:doc("/test/tde.json”)/content/array-node(“children”)/object-node()[1]/text(“name”), fn:doc("/test/tde.json”)/content/array-node(“children”)/object-node()[2]/text(“name”)) cast as xs:string?

What is the proper syntax for extracting array nodes into a triple?

2nd somewhat related question: say I also wanted to have triples of the form “child1 is-sibling-of child2”. For the example above it would be “Bob Child is-sibling-of Sue Child”. What would be the proper syntax for this? I’m not even sure how to begin with this one.

Is TDE even the way to go here? Or is it better to do this programmatically? i.e. on document ingestion, generate those triples inside the document directly?

(If it’s relevant, the ML version being used is 9.)

I’ve been testing migrating one of our systems to Marklogic 9 and using the Optics API.

One of our functions involves grouping claims by member_id, member_name and getting the sums and counts, so I did something like this:

var results = op.fromView('test', 'claims')
  .groupBy(['member_id', 'member_name'], [
         op.count('num_claims', 'claim_no'),
         op.sum('total_amount', 'claim_amount')
         ])
  .orderBy(op.desc('total_amount'))
  .limit(200)
  .result()
  .toArray();

Above works fine. The results are of the form

[
  { 
    member_id: 1, 
    member_name: 'Bob', 
    num_claims: 10, 
    total_amount: 500
  }, 
  ...
]

However, we also have a field “company”, where each claim is filed under a different company. Basically the relevant view columns are claim_no, member_id, member_name, company, claim_amount

I would like to be able to show a column that list the different companies for which the member_id/member_name has filed claims, and how many claims for each company.

i.e. I want my results to be something like:

[
  { 
    member_id: 1, 
    member_name: 'Bob', 
    num_claims: 10, 
    total_amount: 500,
    companies: [
      {
        company: 'Ajax Co',
        num_claims: 8
      },
      {
        company: 'Side Gig',
        num_claims: 2
      }
    ]
  }, 
  ...
]

I tried something like this:

results = results.map((member, index, array) => {
  var companies = op.fromView('test', 'claims')
    .where(op.eq(op.col('member_id'), member.member_id))
    .groupBy('company', [
      op.count('num_claims', 'claim_no')      
    ])
    .result()
    .toArray();
  member.companies = companies;
  return member;
});

And the output seems correct, but it also executes quite slowly - almost a minute (total number of claim documents is around 120k)

In our previous ML8 implementation, we were pre-generating summary documents for each member - so retrieval was reasonably fast with the downside that whenever we got a bunch of new data, all of the summary documents had to be re-generated. I was hoping that ML9’s optic API would make it easier to do the retrieval/grouping/aggregates on the fly so we wouldn’t have to do that.

In theory, I could just add company to the groupBy fields, then merge the rows in the result query as needed. But the problem with that approach is that I can’t guarantee I’ll get the top 200 by total amount (as was my original query)

So, the question is: Is there a better way of doing this with a reasonable execution time? Or should I just stick to pre-generating the summary documents?