Roy Tang

roytang.net

Programmer, engineer, scientist, critic, gamer, dreamer, and kid-at-heart.

Blog Notes Photos Links Archives About

All items tagged marklogic.

Subscribe: RSS JSON

2017 December

  • MarkLogic node.js: stream end event doesn't fire for certain collections

    I have a query using MarkLogic node.js that basically boils down to something like this:

    db.documents.query(qb.where(qb.collection('test'))).stream()
.on('data', function(row) {
    console.log("Stream on data");
})
.on('end', function() {
    console.log("Stream on end");
})
.on('error', function(error) {
    console.log(error);
})
;

    Now, for a certain collection we have in our database, the 'end' function doesn't fire, i.e. I never see "Stream on end" appear in the log. There's no error or anything, processing just stops. It's only for this particular collection, other collections seem fine.

    If I query documents in that collection directly using other methods such as qb.value() without using qb.collection(), the end event fires correctly. But once I add qb.collection() into the mix (using qb.and), the end event doesn't fire.

    I'm unsure how to debug this, as this is my first time trying to use streams in the nodejs client library. Any advice as to what I can check?

    Thanks!

    Posted by under notes at #nodejs #marklogic #questions #stackoverflow #software-development

    Also on: stackexchange /

  • Marklogic Template Driven Extraction: How to define nullable in Javascript template?

    How to specify that a column in the schema should be nullable?

    I tried adding a nullable attribute:

    var myFirstTDE = xdmp.toJSON(
  {
    "template": {
      "context": "/match",
      "collections": ["source1"],
      "rows": [
        {
          "schemaName": "soccer",
          "viewName": "matches",
          "columns": [
            {
              "name": "id",
              "scalarType": "long",
              "val": "id",
              "nullable": 0
            },
            {
              "name": "document",
              "scalarType": "string",
              "val": "docUri"
            },
            {
              "name": "date",
              "scalarType": "date",
              "val": "match-date"
            },
            {
              "name": "league",
              "scalarType": "string",
              "val": "league"
            }
          ]
        }
      ]
    }
  }
);

tde.validate( 
  [myFirstTDE]
);

    But this gave me a template error:

    "message": "TDE-INVALIDTEMPLATENODE: Invalid extraction template node: fn:doc('')/template/array-node('rows')/object-node()/array-node('columns')/object-node()[1]/number-node('nullable')"

    For a template defined using XQuery, adding nullable to the column works:

    <column>
  <name>ISSN</name>
  <scalar-type>string</scalar-type>
  <val>Journal/ISSN</val>
  <nullable>true</nullable>
</column>

    How to do the same thing using JS/Json?

    Posted by under notes at #javascript #marklogic #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 1

2017 November

  • MarkLogic template driven extraction and triples: forming triples between array nodes

    This is a follow-up to my question here: https://stackoverflow.com/questions/47449002/marklogic-template-driven-extraction-and-triples-dealing-with-array-nodes/47459250#47459250

    So let's say I have a number of documents structured like this:

    declareUpdate();

xdmp.documentInsert(
       '/test/tde.json',
       {
         content: {
           name:'Joe Parent',
           children: [
             {
               name: 'Bob Child'
             },
             {
               name: 'Sue Child'
             },
             {
               name: 'Guy Child'
             }
           ]
         }
       },
       {permissions : xdmp.defaultPermissions(),
        collections : ['test']})

    I want to define a template that would extract triples from these documents defining sibling relationships between the children. For the above example, I would want to extract the following triples (the relationship is two-way):

    Bob Child sibling-of Sue Child
Bob Child sibling-of Guy Child
Sue Child sibling-of Bob Child
Sue Child sibling-of Guy Child
Guy Child sibling-of Bob Child
Guy Child sibling-of Sue Child

    How can i set up my template to accomplish this?

    Thanks!

    Posted by under notes at #marklogic #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 2

  • MarkLogic template driven extraction and triples: dealing with array nodes

    I've been studying the examples here: https://docs.marklogic.com/guide/semantics/tde#id_25531

    I have a set of documents that are structured with a parent name and an array of children nodes with their own names. I want to create a template that generates triples of the form "name1 is-a-parent-of name2". Here's a test I tried, with a sample of the document structure:

    declareUpdate();

xdmp.documentInsert(
       '/test/tde.json',
       {
         content: {
           name:'Joe Parent',
           children: [
             {
               name: 'Bob Child'
             },
             {
               name: 'Sue Child'
             }
           ]
         }
       },
       {permissions : xdmp.defaultPermissions(),
        collections : ['test']})

cts.doc('/test/tde.json')

var tde = require("/MarkLogic/tde.xqy");
// Load the user template for user profile rows
var template = xdmp.toJSON(
{
  "template":{
    "context":"content",
    "collections": [
      "test"
    ],
      "triples":[
      {
        "subject": {
          "val": "xs:string(name)"
        },
        "predicate": {
          "val": "sem:iri('is-parent-of')"
        },
        "object": {
          "val": "xs:string(children/name)"
        }     
      }
    ]   
  }
}
);
//tde.validate([template]),
tde.templateInsert("/templates/test.tde", template);
tde.nodeDataExtract( 
  [cts.doc( '/test/tde.json' )]
)

    However, the above throws an Exception:

    [javascript] TDE-EVALFAILED: tde.nodeDataExtract([cts.doc("/test/tde.json")]) -- Eval for Object='xs:string(children/name)' returns TDE-BADVALEXPRESSION: Invalid val expression: XDMP-CAST: (err:FORG0001) Invalid cast: (fn:doc("/test/tde.json")/content/array-node("children")/object-node()[1]/text("name"), fn:doc("/test/tde.json")/content/array-node("children")/object-node()[2]/text("name")) cast as xs:string?

    What is the proper syntax for extracting array nodes into a triple?

    2nd somewhat related question: say I also wanted to have triples of the form "child1 is-sibling-of child2". For the example above it would be "Bob Child is-sibling-of Sue Child". What would be the proper syntax for this? I'm not even sure how to begin with this one.

    Is TDE even the way to go here? Or is it better to do this programmatically? i.e. on document ingestion, generate those triples inside the document directly?

    (If it's relevant, the ML version being used is 9.)

    Posted by under notes at #marklogic #marklogic-9 #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 1

  • Marklogic Optic API

    I've been testing migrating one of our systems to Marklogic 9 and using the Optics API.

    One of our functions involves grouping claims by member_id, member_name and getting the sums and counts, so I did something like this:

    var results = op.fromView('test', 'claims')
  .groupBy(['member_id', 'member_name'], [
         op.count('num_claims', 'claim_no'),
         op.sum('total_amount', 'claim_amount')
         ])
  .orderBy(op.desc('total_amount'))
  .limit(200)
  .result()
  .toArray();

    Above works fine. The results are of the form 

    [
  { 
    member_id: 1, 
    member_name: 'Bob', 
    num_claims: 10, 
    total_amount: 500
  }, 
  ...
]

    However, we also have a field "company", where each claim is filed under a different company. Basically the relevant view columns are claim_no, member_id, member_name, company, claim_amount

    I would like to be able to show a column that list the different companies for which the member_id/member_name has filed claims, and how many claims for each company.

    i.e. I want my results to be something like:

    [
  { 
    member_id: 1, 
    member_name: 'Bob', 
    num_claims: 10, 
    total_amount: 500,
    companies: [
      {
        company: 'Ajax Co',
        num_claims: 8
      },
      {
        company: 'Side Gig',
        num_claims: 2
      }
    ]
  }, 
  ...
]

    I tried something like this:

    results = results.map((member, index, array) => {
  var companies = op.fromView('test', 'claims')
    .where(op.eq(op.col('member_id'), member.member_id))
    .groupBy('company', [
      op.count('num_claims', 'claim_no')      
    ])
    .result()
    .toArray();
  member.companies = companies;
  return member;
});

    And the output seems correct, but it also executes quite slowly - almost a minute (total number of claim documents is around 120k)

    In our previous ML8 implementation, we were pre-generating summary documents for each member - so retrieval was reasonably fast with the downside that whenever we got a bunch of new data, all of the summary documents had to be re-generated. I was hoping that ML9's optic API would make it easier to do the retrieval/grouping/aggregates on the fly so we wouldn't have to do that.

    In theory, I could just add company to the groupBy fields, then merge the rows in the result query as needed. But the problem with that approach is that I can't guarantee I'll get the top 200 by total amount (as was my original query)

    So, the question is: Is there a better way of doing this with a reasonable execution time? Or should I just stick to pre-generating the summary documents?

    Posted by under notes at #marklogic #marklogic-9 #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 2

  • Marklogic Roxy: Calling a javascript module from app_specific.rb

    I have a Marklogic 9 project that I'm configuring with Roxy. I've been following these examples: https://github.com/marklogic-community/roxy/wiki/Adding-Custom-Build-Steps

    Basically, I have a server-side JS function that I want to call after deploy content. I have something like this:

    # then you would define your new method

      def deploy_content
    # you can optionally call the original
    original_deploy_content

    # do your stuff here
    execute_query(%Q{
      xquery version "1.0-ml";
      xdmp:javascript-eval('var process = require("/ingestion/process.sjs"); process.postDeployContent();')
    },
    :db_name => @properties["ml.app-name"] + "-content")

  end

    The xquery being called here evaluates fine when executed via query console. But when I call ml local deploy content, I get the following error:

    ERROR: 500 "Internal Server Error"
ERROR: <html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>500 Internal Server Error</title>
    <meta name="robots" content="noindex,nofollow"/>
    <link rel="stylesheet" href="/error.css"/>
  </head>
  <body>
    <span class="error">
      <h1>500 Internal Server Error</h1>
      <dl>
        <dt>XDMP-MODNOTFOUND: var process = require("/ingestion/process.sjs"); process.postDeployContent(); -- Module /ingestion/process.sjs not found</dt>
        <dd></dd>
        <dt>in [anonymous], at 1:14 [javascript]</dt>
        <dd></dd>
        <dt>at 3:6,
in xdmp:eval("var process = require(&amp;quot;/ingestion/process.sjs&amp;quot;); proce...") [javascript]</dt>
        <dd></dd>
        <dt>in /eval, at 3:6 [1.0-ml]</dt>
        <dd></dd>
      </dl>
    </span>
  </body>
</html>

    Why is the module not found when running via xquery from app_specific.rb?

    Or... is there a better way to call a JS module function from here. Sorry, I'm not too familiar with the xquery side, so I just called a JS function instead.

    Posted by under notes at #marklogic #roxy #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 1

2017 October

  • Marklogic 9 + Roxy: can't connect to created database using Node.js

    I'm trying out the Roxy deployer. The Roxy app was created using the default app-type. I setup a new ML 9 database, and I ran "ml local bootstrap" using the default ports (8040 and 8041)

    Then I setup a node application. I tried the following (sample code from https://docs.marklogic.com/jsdoc/index.html)

    var marklogic = require('marklogic');
var conn = {
    host: '192.168.33.10',  
    port: 8040,
    user: 'admin',
    password: 'admin',
    authType: 'DIGEST'
}

var db = marklogic.createDatabaseClient(conn);

db.createCollection(
  '/books',
  {author: 'Beryl Markham'},
  {author: 'WG Sebald'}
  )
.result(function(response) {
    console.log(JSON.stringify(response, null, 2));
  }, function (error) {
    console.log(JSON.stringify(error, null, 2));
  });

    Running the script gave me an error like:

    $ node test.js
{
  "message": "write document list: cannot process response with 500 status",
  "statusCode": 500,
  "body": "<error:error xsi:schemaLocation=\"http://marklogic.com/xdmp/error error.xsd\" xmlns:error=\"http://marklogic.com/xdmp/error\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n  <error:code>XDMP-IMPMODNS</error:code>\n  <error:name>err:XQST0059</error:name>\n  <error:xquery-version>1.0-ml</error:xquery-version>\n  <error:message>Import module namespace mismatch</error:message>\n  <error:format-string>XDMP-IMPMODNS: (err:XQST0059) Import module namespace http://marklogic.com/rest-api/endpoints/config does not match target namespace http://marklogic.com/rest-api/endpoints/config_DELETE_IF_UNUSED of imported module /MarkLogic/rest-api/endpoints/config.xqy</error:format-string>\n  <error:retryable>false</error:retryable>\n  <error:expr/>\n  <error:data>\n    <error:datum>http://marklogic.com/rest-api/endpoints/config</error:datum>\n    <error:datum>http://marklogic.com/rest-api/endpoints/config_DELETE_IF_UNUSED</error:datum>\n    <error:datum>/MarkLogic/rest-api/endpoints/config.xqy</error:datum>\n  </error:data>\n  <error:stack>\n    <error:frame>\n      <error:uri>/roxy/lib/rewriter-lib.xqy</error:uri>\n      <error:line>5</error:line>\n      <error:column>0</error:column>\n      <error:xquery-version>1.0-ml</error:xquery-version>\n    </error:frame>\n  </error:stack>\n</error:error>\n"
}

    If I change the port to 8000 (the default appserver that inserts into Documents), the node function executes correctly as expected. I'm not sure if I need to configure anything else with the Roxy-created appserver so that it works with the node.js application.

    I'm not sure where the "DELETE_IF_UNUSED" part in the error message is coming from either. There doesn't seem to be any such text in the configuration files generated by Roxy.

    Edit: When accessing 192.168.33.10:8040 via the browser, I get a an xml with a similar error:

    <error:error xsi:schemaLocation="http://marklogic.com/xdmp/error error.xsd" xmlns:error="http://marklogic.com/xdmp/error" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <error:code>XDMP-IMPMODNS</error:code>
  <error:name>err:XQST0059</error:name>
  <error:xquery-version>1.0-ml</error:xquery-version>
  <error:message>Import module namespace mismatch</error:message>
  <error:format-string>XDMP-IMPMODNS: (err:XQST0059) Import module namespace http://marklogic.com/rest-api/endpoints/config does not match target namespace http://marklogic.com/rest-api/endpoints/config_DELETE_IF_UNUSED of imported module /MarkLogic/rest-api/endpoints/config.xqy</error:format-string>
  <error:retryable>false</error:retryable>
  <error:expr/>
  <error:data>
    <error:datum>http://marklogic.com/rest-api/endpoints/config</error:datum>
    <error:datum>http://marklogic.com/rest-api/endpoints/config_DELETE_IF_UNUSED</error:datum>
    <error:datum>/MarkLogic/rest-api/endpoints/config.xqy</error:datum>
  </error:data>
  <error:stack>
    <error:frame>
      <error:uri>/roxy/lib/rewriter-lib.xqy</error:uri>
      <error:line>5</error:line>
      <error:column>0</error:column>
      <error:xquery-version>1.0-ml</error:xquery-version>
    </error:frame>
  </error:stack>
</error:error>

    If it matters, MarkLogic version is 9.0-3.1. It's a fresh install too.

    Any advice?

    Posted by under notes at #nodejs #marklogic #marklogic-9 #roxy #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 3

  • Marklogic 9: MLCP fails when importing records with a transform

    I'm trying to migrate one of my dev envts from ML8 to ML9. I have an import script that successfully works on the ML8 version, but there's an error when I try running it against the ML9 database. The ML9 version is 9.0.3.1. The MLCP version is 9.0.3

    My MLCP options file is as follows:

    import
-host 
192.168.33.10
-port 
8041
-username 
admin
-password
admin
-input_file_path 
d:\maroon\data\mbastest.csv 
-mode 
local 
-input_file_type 
delimited_text 
-uri_id 
ClientId 
-output_uri_prefix
/test/records/
-output_uri_suffix 
.json 
-document_type 
json 
-transform_module 
/ingestion/transform.js
-transform_function 
testTransform
-transform_param
test
-content_encoding 
windows-1252 
-thread_count
1

    Here's the output of a test run with only 2 records in the test CSV file:

    17/10/30 14:07:33 INFO contentpump.LocalJobRunner: Content type: JSON
17/10/30 14:07:33 INFO contentpump.ContentPump: Job name: local_455168344_1
17/10/30 14:07:33 INFO contentpump.FileAndDirectoryInputFormat: Total input paths to process : 1
17/10/30 14:07:38 WARN contentpump.TransformWriter: Failed document /test/records/31.json
17/10/30 14:07:38 WARN contentpump.TransformWriter: <error:format-string xmlns:error="http://marklogic.com/xdmp/error" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected QName_, expecting $end or SemiColon_</error:format-string>
17/10/30 14:07:38 WARN contentpump.TransformWriter: Failed document /test/records/32.json
17/10/30 14:07:38 WARN contentpump.TransformWriter: <error:format-string xmlns:error="http://marklogic.com/xdmp/error" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected QName_, expecting $end or SemiColon_</error:format-string>
17/10/30 14:07:38 INFO contentpump.LocalJobRunner:  completed 100%
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: com.marklogic.mapreduce.MarkLogicCounter:
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: INPUT_RECORDS: 2
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS: 2
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_COMMITTED: 0
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_FAILED: 2
17/10/30 14:07:38 INFO contentpump.LocalJobRunner: Total execution time: 5 sec

    If I remove the transform params, the import works fine.

    I thought it might be a parsing issue with my transform module itself, so I tried replacing it with the following example from the documentation:

    // Add a property named "NEWPROP" to any JSON input document.
// Otherwise, input passes through unchanged.

function addProp(content, context)
{
  const propVal = (context.transform_param == undefined)
                 ? "UNDEFINED" : context.transform_param;

  if (xdmp.nodeKind(content.value) == 'document' &&
      content.value.documentFormat == 'JSON') {
    // Convert input to mutable object and add new property
    const newDoc = content.value.toObject();
    newDoc.NEWPROP = propVal;

    // Convert result back into a document
    content.value = xdmp.unquote(xdmp.quote(newDoc));
  }
  return content;
};

exports.addProp = addProp;

    (Of course I changed the params in the MLCP options file accordingly)

    The issue still persists even with just this test function.

    Any advice?

    Posted by under notes at #marklogic #mlcp #marklogic-9 #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 2

2017 September

2017 August

  • MarkLogic node.js: sorting by the value of a json element with multiple instances in the same document

    Sorry if the title is unclear, but I'm finding my problem hard to explain concisely. I have a number of JSON documents with structure like this:

    {
  "count": 100,
  "groups": [
    {
    "name": "group A",
    "count": 12
    },
    {
    "name": "group B",
    "count": 22
    },
    {
    "name": "group C",
    "count": 7
    }
  ]
}

    Basically, the document has an item count plus a breakdown of that count into smaller groups. So this record represents a collection of 100 items, of which 12 are from group A, 22 are from group B, 7 are from group C.

    Now, I have an element range index on "count" and a bunch of such documents. I'd like to be able to sort by any of the options:

    • sort by total count (descending or ascending)
    • sort by group A count (descending or ascending)
    • sort by group B count (descending or ascending)
    • sort by group C count (descending or ascending)

    I've tried

    .orderBy(qb.sort("count", "descending"))

    This seems to be sort by the total cost (the one in the document root), but I'm not sure if that's always true or I need to specify something else to guarantee it always gets that particular one.

    For sorting by a specific group, I have no idea how to specify it.

    Any advice?

    Posted by under notes at #marklogic #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 2

  • Marklogic - flexible replication

    We've been trying to set up flexible replication in our system which uses a MarkLogic database. We followed the instructions from https://docs.marklogic.com/8.0/guide/flexrep/quick_start and have been able to set up flexible replication between two MarkLogic servers. We have verified that new documents created in the master are copied over to the replica. However, the master database currently has more than 47 million records that were there before we configured the replication. Once the replication process was triggered, we observed that the documents are being replicated to the replica very slowly. Roughly 20,000 documents were replicated within the first two hours. The rate is very slow, it would take months for the old records to be fully replicated.

    Our questions are:

    1. We are looking into increasing the hardware specs of the two servers, but aside from that would anyone have any advice or documentation as to how we could speed up the replication? I couldn't find any existing documentation regarding this?

    2. Failing that, would it be possible to set up flexible replication without needing to replicate the initial data set? FYI, we also tried to clone the master database and use the clone as the replica. (We thought this might mean that the older records don't have to be replicated.) However, in this case we encountered XDMP-NEWSTAMP and XDMP-EXTIME errors on the replica server, so we gave up on this approach. A sample of the errors encountered on the replica is below:

    2017-08-03 18:45:04.376 Notice: exp-rest-content-flexrep: XDMP-NEWSTAMP: Timestamp too new for forest exp-rest-content-001-1 (15017569242290900) 2017-08-03 18:45:04.376 Notice: exp-rest-content-flexrep: in /apply.xqy [1.0-ml] 2017-08-03 18:45:04.379 Notice: TaskServer: XDMP-EXTIME: try { let $raw-module-name := module-path($action-to-execute/p:module) let $module-kind := module-kind($raw-module-name) let $module-name := if ($module-kind = "xquery" or $module-kind = "javascript") then $raw-module-name else $cpfi:xslt-action return if ($module-name = "") then fn:error((), "CPF-ACTIONNOTFOUND", "Default success") else if ($module-kind = "javascript") then (xdmp:trace("CPF Action Invoke", fn:string-join(($caller, xdmp:get-current-user(), $uri, $state-or-status, $raw-module-name), " ")), xdmp:invoke($module-name, (fn:QName("","uri"), $uri, xs:QName("cpf:document-uri"), $uri, fn:QName("","transition"), $chosen-transition, options-var-js($action-to-execute)), $invoke-options)) else (xdmp:trace("CPF Action Invoke", fn:string-join(($caller, xdmp:get-current-user(), $uri, $state-or-status, $raw-module-name), " ")), xdmp:invoke($module-name, ($vars, xs:QName("cpf:transition"), $chosen-transition, options-var($action-to-execute), if ($module-kind = "xslt") then (xs:QName("cpf:stylesheet-uri"), $raw-module-name) else ()), $invoke-options)) } catch ($e) { let $trace := let $context := fn:concat($caller, " ", $uri, " action failed") return (cpf:log(fn:string-join(($context, $e/err:format-string), " "), "error"), cpf:log(($context, $e), "fine")) let $failure-action := ($pipelines/p:failure-action)[1] let $raw-failure-module := module-path($failure-action/p:module) let $failure-kind := module-kind($raw-failure-module) let $failure-module := if ($failure-kind = "xquery" or $failure-kind = "javascript") then $raw-failure-module else $cpfi:xslt-action return if ($failure-module = "") then fn:error((), "CPF-ACTIONNOTFOUND", "Default failure action") else xdmp:invoke($failure-module, ($vars, xs:QName("cpf:transition"), $chosen-transition, options-var($failure-action), xs:QName("cpf:exception"), $e, if ($failure-kind = "xslt") then (xs:QName("cpf:stylesheet-uri"), $raw-failure-module) else ()), $invoke-options) } -- Time limit exceeded 2017-08-03 18:45:04.379 Notice: TaskServer: in /MarkLogic/cpf/triggers/internal-cpf.xqy, at 213:4, 2017-08-03 18:45:04.379 Notice: TaskServer: in execute-action("on-state-enter", "http://marklogic.com/states/initial", "/_smslogs/5849823.xml", (xs:QName("trgr:uri"), "/_smslogs/5849823.xml", xs:QName("trgr:trigger"), ...), different-transactiont..., (fn:doc("http://marklogic.com/cpf/pipelines/12349495875628658916.xml")/p:pipeline, fn:doc("http://marklogic.com/cpf/pipelines/3358424510998587926.xml")/p:pipeline, fn:doc("http://marklogic.com/cpf/pipelines/13179541037342910978.xml")/p:pipeline, ...), fn:doc("http://marklogic.com/cpf/pipelines/3358424510998587926.xml")/p:pipeline/p:state-transition[3]/p:default-action, fn:doc("http://marklogic.com/cpf/pipelines/3358424510998587926.xml")/p:pipeline/p:state-transition[3]) [1.0-ml] 2017-08-03 18:45:04.379 Notice: TaskServer: $caller = "on-state-enter" 2017-08-03 18:45:04.379 Notice: TaskServer:
    $state-or-status = "http://marklogic.com/states/initial" 2017-08-03 18:45:04.379 Notice: TaskServer: $uri = "/_smslogs/5849823.xml" 2017-08-03 18:45:04.379 Notice: TaskServer: $vars = (xs:QName("trgr:uri"), "/_smslogs/5849823.xml", xs:QName("trgr:trigger"), ...) 2017-08-03 18:45:04.379 Notice: TaskServer: $invoke-options = different-transactiont... 2017-08-03 18:45:04.379 Notice: TaskServer: $pipelines = (fn:doc("http://marklogic.com/cpf/pipelines/12349495875628658916.xml")/p:pipeline, fn:doc("http://marklogic.com/cpf/pipelines/3358424510998587926.xml")/p:pipeline, fn:doc("http://marklogic.com/cpf/pipelines/13179541037342910978.xml")/p:pipeline, ...) 2017-08-03 18:45:04.379 Notice: TaskServer: $action-to-execute = fn:doc("http://marklogic.com/cpf/pipelines/3358424510998587926.xml")/p:pipeline/p:state-transition[3]/p:default-action 2017-08-03 18:45:04.379 Notice: TaskServer: $chosen-transition = fn:doc("http://marklogic.com/cpf/pipelines/3358424510998587926.xml")/p:pipeline/p:state-transition[3] 2017-08-03 18:45:04.379 Notice: TaskServer: $e = xsi:schemaLocation="http://marklogic.com/xdmp/error error.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:error="http://marklogic.com/xdmp/error">XDMP-NEWSTAMP<error:xquery... 2017-08-03 18:45:04.379 Notice: TaskServer: in /MarkLogic/cpf/triggers/internal-cpf.xqy, at 342:6, 2017-08-03 18:45:04.379 Notice: TaskServer: in execute-transition("on-state-enter", "http://marklogic.com/states/initial", "/_smslogs/5849823.xml", (xs:QName("trgr:uri"), "/_smslogs/5849823.xml", xs:QName("trgr:trigger"), ...), xmlns:trgr="http://marklogic.com/xdmp/triggers">6551367241994447650<trgr:trig..., (fn:doc("http://marklogic.com/cpf/pipelines/12349495875628658916.xml")/p:pipeline, fn:doc("http://marklogic.com/cpf/pipelines/3358424510998587926.xml")/p:pipeline, fn:doc("http://marklogic.com/cpf/pipelines/13179541037342910978.xml")/p:pipeline, ...), (fn:doc("http://marklogic.com/cpf/pipelines/12349495875628658916.xml")/p:pipeline/p:state-transition[2], fn:doc("http://marklogic.com/cpf/pipelines/3358424510998587926.xml")/p:pipeline/p:state-transition[3], fn:doc("http://marklogic.com/cpf/pipelines/13179541037342910978.xml")/p:pipeline/p:state-transition[1], ...), xmlns:p="http://marklogic.com/cpf/pipelines"><p:state>http://marklogic.com/states/initial) [1.0-ml] 2017-08-03 18:45:04.379 Notice: TaskServer: $caller = cpf:state("http://marklogic.com/states/initial") 2017-08-03 18:45:04.379 Notice: TaskServer: $state-or-status = () 2017-08-03 18:45:04.379 Notice: TaskServer: $uri = (xs:QName("trgr:uri"), "/_smslogs/5849823.xml", xs:QName("trgr:trigger"), ...) 2017-08-03 18:45:04.379 Notice: TaskServer: in /MarkLogic/cpf/triggers/internal-cpf.xqy, at 358:3, 2017-08-03 18:45:04.379 Notice: TaskServer: in int:execute-state-transition("on-state-enter", cpf:state("http://marklogic.com/states/initial"), "/_smslogs/5849823.xml", (xs:QName("trgr:uri"), "/_smslogs/5849823.xml", xs:QName("trgr:trigger"), ...), xmlns:trgr="http://marklogic.com/xdmp/triggers">6551367241994447650<trgr:trig...) [1.0-ml] 2017-08-03 18:45:04.379 Notice: TaskServer: $caller = cpf:state("http://marklogic.com/states/initial") 2017-08-03 18:45:04.379 Notice: TaskServer: $state = () 2017-08-03 18:45:04.379 Notice: TaskServer: $uri = (xs:QName("trgr:uri"), "/_smslogs/5849823.xml", xs:QName("trgr:trigger"), ...) 2017-08-03 18:45:04.379 Notice: TaskServer: in /MarkLogic/cpf/triggers/on-state-enter.xqy, at 41:6 [1.0-ml] 2017-08-03 18:45:04.379 Notice: TaskServer: $state = cpf:state("http://marklogic.com/states/initial") 2017-08-03 18:45:04.379 Notice: TaskServer: $trace = () 2017-08-03 18:45:04.379 Notice: TaskServer: $vars = (xs:QName("trgr:uri"), "/_smslogs/5849823.xml", xs:QName("trgr:trigger"), ...) 2017-08-03 18:45:04.379 Notice: TaskServer: XDMP-NEWSTAMP: Timestamp too new for forest exp-rest-content-001-1 (15017569242290900) 2017-08-03 18:45:04.379 Notice: exp-rest-content-flexrep: XDMP-NEWSTAMP: Timestamp too new for forest exp-rest-content-001-1 (15017569242290900) 2017-08-03 18:45:04.379 Notice: exp-rest-content-flexrep: in /apply.xqy [1.0-ml]

    Posted by under notes at #marklogic #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 1

  • MarkLogic node.js - is it possible to support a derived value in parseBindings?

    We have a search application using MarkLogic node.js. We use parsedQuery like this:

                qb.parsedFrom(prop.search, 
            qb.parseBindings(
                qb.word('name', qb.bind('name')),
                qb.word('birthdate', qb.bind('birthdate')),
                qb.range('count', qb.datatype('float'), qb.bind('count'))
            )
        )

    The above currently supports search syntax like "count GT 50", etc. We would like to support searching using a derived value such as age. That is, we want to support a search syntax like "age GT 10", where the age value is not stored in the documents in the database but rather needs to be computed from the birthdate on the fly. We can't store the age in the documents since the age changes depending on the current date.

    Is this possible and if so, how? If it matters, we are using ML8

    Posted by under notes at #marklogic #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 2

2017 July

  • MarkLogic - detecting similar/duplicate names

    I have a number of documents from different sources. Many of them reference a company name, but may have stored the information slightly differently. The name is a field in the documents.

    I'd like to be able to detect variations on the same name, something like:

    • Ajax Company Incorporated
    • Ajax Co. Inc.
    • Ajax Company Inc.
    • Ajax Company
    • Ajax Company (formerly Ajax Unlimited)
    • etc

    Does MarkLogic have any facility to query documents that have "similar" name as above? I'm not sure if there's a more technical term that I should be searching for. Preferably for either the node client API or server-side js.

    Posted by under notes at #marklogic #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 1

2017 June

2016 December

  • Marklogic Node.js API: How to get the document where an embedded triple lives?

    I tried to insert the following test document:

    db.documents.write(
    {
        uri: "/test/doc1.json",
        contentType: "application/json",
        collections: "test",
        content: {
            name : "Peter",
            hobby: "Sleeping",
            other: "Some other info",
            "triple": {
                "subject": {   
                    "datatype": "http://example.com/name/",  
                    "value": "Peter"   
                },   
                "predicate": {     
                    "datatype": "http://example.com/relation/",  
                    "value": "livesin"   
                },   
                "object": {     
                    "datatype": "http://example.com/location/",  
                    "value": "Paris"   
                }
            }
        }
    }
  ).
  result(function(response){
    console.log("Done loading");
  });

    Then I queried as follows:

    var query = [
  'SELECT ?s ?p ?o' ,
  'WHERE { ?s ?p ?o }' ,
];
db.graphs.sparql('application/sparql-results+json', query.join('\n')
).result(function (result) {
  console.log(JSON.stringify(result, null, 2));
}, function(error) {
  console.log(JSON.stringify(error, null, 2));
});

    The results showed me the values of the triple, but what if I also want to get the entire document where the triple was embedded? Is it also possible to filter by other fields in the document?

    Posted by under notes at #nodejs #sparql #marklogic #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 2

  • How to query Marklogic triples using SPARQL?

    I inserted a test document using the Node.js API:

      db.documents.write(
    {
        uri: "/test/doc1.json",
        contentType: "application/json",
        collections: "test",
        content: {
            name : "Peter",
            "triple": {
                "subject": {   
                    "datatype": "http://example.com/name/",  
                    "value": "Peter"   
                },   
                "predicate": {     
                    "datatype": "http://example.com/relation/",  
                    "value": "livesin"   
                },   
                "object": {     
                    "datatype": "http://example.com/location/",  
                    "value": "Paris"   
                }
            }
        }
    }
  ).
  result(function(response){
    console.log("Done loading");
  });

    Then I tried querying it via SPARQL on the Marklogic console webapp:

    SELECT ?s ?p ?o
  WHERE { ?s ?p ?o }

    The query above works fine, it retrieves the test triple I inserted above

    However, if I want to filter by the literal value "Paris", I tried the following:

    PREFIX loc: <http://example.com/location/>

SELECT ?s ?p 
  WHERE { ?s ?p loc:Paris }

    But in this case I got zero results. Is this syntax incorrect? I was just following how the queries looked in https://docs.marklogic.com/guide/semantics/semantic-searches

    What is the proper way to do it?

    Posted by under notes at #sparql #marklogic #questions #stackoverflow #software-development

    Also on: stackexchange / 💬 1

2016 November

On This Site
Things I Write About
Elsewhere
Subscribe
Webrings