Scala
  1. Scala
  2. SCALA-55

Map Reduce works from command line, fails w/ Casbah

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major - P3 Major - P3
    • Resolution: Works as Designed
    • Affects Version/s: casbah-2.1.2
    • Fix Version/s: None
    • Component/s: Casbah Core
    • Labels:
    • Environment:
    • Operating System:
      Linux
    • # Replies:
      9
    • Last comment by Customer:
      true

      Description

      Apologies in advance if this is something obvious.

      Error:
      Mon Sep 26 18:59:27 [conn34] mr ns: jayed.tweets
      Mon Sep 26 18:59:27 [conn34] query jayed.system.js reslen:20 0ms
      Mon Sep 26 18:59:27 [conn34] JS Error: SyntaxError: missing : after property id nofile_b:0
      Mon Sep 26 18:59:27 [conn34] compile failed for: function (){ emit(

      { this.topic }, { count: 1 }); }
      Mon Sep 26 18:59:27 [conn34] User Assertion: 13598:couldn't compile code for: _map
      Mon Sep 26 18:59:27 [conn34] mr failed, removing collection :: caused by :: 13598 couldn't compile code for: _map
      Mon Sep 26 18:59:27 [conn34] run command jayed.$cmd { drop: "tmp.mr.tweets_24" }
      Mon Sep 26 18:59:27 [conn34] CMD: drop jayed.tmp.mr.tweets_24
      Mon Sep 26 18:59:27 [conn34] command jayed.$cmd command: { drop: "tmp.mr.tweets_24" } ntoreturn:1 reslen:62 0ms
      Mon Sep 26 18:59:27 [conn34] run command jayed.$cmd { drop: "tmp.mr.tweets_24_inc" }
      Mon Sep 26 18:59:27 [conn34] CMD: drop jayed.tmp.mr.tweets_24_inc
      Mon Sep 26 18:59:27 [conn34] command jayed.$cmd command: { drop: "tmp.mr.tweets_24_inc" } ntoreturn:1 reslen:62 0ms

      Sample code
      val coll = mongo("jayed")("tweets")
      val map = "function (){ emit({ this.topic }

      ,

      { count: 1 }

      ); }"
      val reduce = "function(key, values) { var count = 0; values.forEach(function(v)

      { count += v['count']; }); return {count: count}; }"
      val out = coll.mapReduce( map , reduce , "tweetresults" , Some(MongoDBObject( "verbose" -> true )) )
      val tweetresults = mongo("jayed")("tweetresults")
      tweetresults.ensureIndex( MongoDBObject("count" -> 1 ) , "topic_count_index", false )
      tweetresults.find().toString() //is an empty iterator

      Tried also with MapReduceInlineOutput.

      Given a schema/data like
      {"topic" : "A" , "comment" : "hello world" } , {"topic": null , "comment" : "hello again" } , {"topic" : "A" , "comment" : "hello world finally" }



      Following works from command line

      map = function (){
      emit({ this.topic }, { count: 1 });
      }

      reduce = function(key, values) { var count = 0; values.forEach(function(v) { count += v['count']; }

      ); return

      {count: count}

      ; };

      db.tweets.mapReduce( map, reduce,

      { out: "tweetresults" }

      ); //
      db.tweetresults.ensureIndex(

      {count : 1}

      );
      db.tweetresults.find().sort(

      {count : 1}

      );

      One result for "topic" above is null as a value FWIW.

      Don't know if this affects 2.1.2 and/or 2.2, but Casbah version 2.1.5

      For some further background:

      http://stackoverflow.com/questions/7302300/group-by-aggregate-map-reduce-functions-in-mongodb-using-scala-casbah-rogue

        Activity

        Hide
        Jonathan Hendler
        added a comment -

        val map = "function (){ emit(

        { topic: this.topic }

        ,

        { count: 1 }

        ); }" fixes the output warning.
        Still, no results are returned.

        Confused in the logs by , verbose: false...

        The new log messages are:

        Mon Sep 26 19:24:59 [initandlisten] connection accepted from 127.0.0.1:37821 #38
        Mon Sep 26 19:24:59 [conn38] run command admin.$cmd

        { isMaster: 1 }

        Mon Sep 26 19:24:59 [conn38] command admin.$cmd command:

        { isMaster: 1 }

        ntoreturn:1 reslen:71 0ms
        Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { mapreduce: "tweets", map: "function (){ emit(

        { topic: this.topic }

        ,

        { count: 1 }

        ); }", reduce: "function(key, values) { var count = 0; values.forEach(function(v)

        { count += v['count']; }); return {count: count}; }", verbose: false, out: "tweetresults", query: { verbose: true } }
        Mon Sep 26 19:24:59 [conn38] mr ns: jayed.tweets
        Mon Sep 26 19:24:59 [conn38] query jayed.system.js reslen:20 0ms
        Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tmp.mr.tweets_26_inc" }
        Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tmp.mr.tweets_26_inc
        Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tmp.mr.tweets_26_inc" } ntoreturn:1 reslen:62 0ms
        Mon Sep 26 19:24:59 [conn38] create collection jayed.tmp.mr.tweets_26_inc { autoIndexId: 0 }
        Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26_inc
        Mon Sep 26 19:24:59 [conn38] build index jayed.tmp.mr.tweets_26_inc { 0: 1 }
        Mon Sep 26 19:24:59 [conn38] external sort root: /var/lib/mongodb/_tmp/esort.1317079499.454318894/
        Mon Sep 26 19:24:59 [conn38] external sort used : 0 files in 0 secs
        Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26_inc.$0_1
        Mon Sep 26 19:24:59 [conn38] done building bottom layer, going to commit
        Mon Sep 26 19:24:59 [conn38] fastBuildIndex dupsToDrop:0
        Mon Sep 26 19:24:59 [conn38] build index done 0 records 0 secs
        Mon Sep 26 19:24:59 [conn38] insert jayed.system.indexes 0ms
        Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tmp.mr.tweets_26" }
        Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tmp.mr.tweets_26
        Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tmp.mr.tweets_26" } ntoreturn:1 reslen:62 0ms
        Mon Sep 26 19:24:59 [conn38] create collection jayed.tmp.mr.tweets_26 {}
        Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26
        Mon Sep 26 19:24:59 [conn38] adding _id index for collection jayed.tmp.mr.tweets_26
        Mon Sep 26 19:24:59 [conn38] build index jayed.tmp.mr.tweets_26 { _id: 1 }
        Mon Sep 26 19:24:59 [conn38] external sort root: /var/lib/mongodb/_tmp/esort.1317079499.1955500750/
        Mon Sep 26 19:24:59 [conn38] external sort used : 0 files in 0 secs
        Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26.$id
        Mon Sep 26 19:24:59 [conn38] done building bottom layer, going to commit
        Mon Sep 26 19:24:59 [conn38] fastBuildIndex dupsToDrop:0
        Mon Sep 26 19:24:59 [conn38] build index done 0 records 0 secs
        Mon Sep 26 19:24:59 [conn38] query jayed.system.indexes nreturned:3 reslen:258 0ms
        Mon Sep 26 19:24:59 [conn38] build index jayed.tmp.mr.tweets_26 { count: -1.0 }
        Mon Sep 26 19:24:59 [conn38] external sort root: /var/lib/mongodb/_tmp/esort.1317079499.541709764/
        Mon Sep 26 19:24:59 [conn38] external sort used : 0 files in 0 secs
        Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26.$count_-1
        Mon Sep 26 19:24:59 [conn38] done building bottom layer, going to commit
        Mon Sep 26 19:24:59 [conn38] fastBuildIndex dupsToDrop:0
        Mon Sep 26 19:24:59 [conn38] build index done 0 records 0 secs
        Mon Sep 26 19:24:59 [conn38] build index jayed.tmp.mr.tweets_26 { count: 1.0 }
        Mon Sep 26 19:24:59 [conn38] external sort root: /var/lib/mongodb/_tmp/esort.1317079499.1868679281/
        Mon Sep 26 19:24:59 [conn38] external sort used : 0 files in 0 secs
        Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26.$count_1
        Mon Sep 26 19:24:59 [conn38] done building bottom layer, going to commit
        Mon Sep 26 19:24:59 [conn38] fastBuildIndex dupsToDrop:0
        Mon Sep 26 19:24:59 [conn38] build index done 0 records 0 secs
        Mon Sep 26 19:24:59 [conn38] query jayed.system.indexes nreturned:1 reslen:98 0ms
        Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tweetresults" }
        Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tweetresults
        Mon Sep 26 19:24:59 [conn38] dropCollection: jayed.tweetresults
        Mon Sep 26 19:24:59 [conn38] dropIndexes done
        Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tweetresults" } ntoreturn:1 reslen:125 0ms
        Mon Sep 26 19:24:59 [conn38] run command admin.$cmd { renameCollection: "jayed.tmp.mr.tweets_26", to: "jayed.tweetresults" }
        Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tweetresults
        Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tweetresults.$id
        Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tweetresults.$count_-1
        Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tweetresults.$count_1
        Mon Sep 26 19:24:59 [conn38] command admin.$cmd command: { renameCollection: "jayed.tmp.mr.tweets_26", to: "jayed.tweetresults" } ntoreturn:1 reslen:37 1ms
        Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tmp.mr.tweets_26" }
        Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tmp.mr.tweets_26
        Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tmp.mr.tweets_26" } ntoreturn:1 reslen:62 0ms
        Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tmp.mr.tweets_26" }
        Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tmp.mr.tweets_26
        Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tmp.mr.tweets_26" } ntoreturn:1 reslen:62 0ms
        Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tmp.mr.tweets_26_inc" }
        Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tmp.mr.tweets_26_inc
        Mon Sep 26 19:24:59 [conn38] dropCollection: jayed.tmp.mr.tweets_26_inc
        Mon Sep 26 19:24:59 [conn38] dropIndexes done
        Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tmp.mr.tweets_26_inc" } ntoreturn:1 reslen:133 0ms
        Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { mapreduce: "tweets", map: "function (){ emit({ topic: this.topic }, { count: 1 }); }", reduce: "function(key, values) { var count = 0; values.forEach(function(v) { count += v['count']; }

        ); return

        {count: count}

        ; }", verbose: false, out: "tweetresults", query:

        { verbose: true }

        } ntoreturn:1 reslen:136 13ms
        Mon Sep 26 19:24:59 [conn38] insert jayed.system.indexes 0ms
        Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd

        { getlasterror: 1, w: 1, wtimeout: 0 }

        Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command:

        { getlasterror: 1, w: 1, wtimeout: 0 }

        ntoreturn:1 reslen:78 0ms

        Show
        Jonathan Hendler
        added a comment - val map = "function (){ emit( { topic: this.topic } , { count: 1 } ); }" fixes the output warning. Still, no results are returned. Confused in the logs by , verbose: false... The new log messages are: Mon Sep 26 19:24:59 [initandlisten] connection accepted from 127.0.0.1:37821 #38 Mon Sep 26 19:24:59 [conn38] run command admin.$cmd { isMaster: 1 } Mon Sep 26 19:24:59 [conn38] command admin.$cmd command: { isMaster: 1 } ntoreturn:1 reslen:71 0ms Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { mapreduce: "tweets", map: "function (){ emit( { topic: this.topic } , { count: 1 } ); }", reduce: "function(key, values) { var count = 0; values.forEach(function(v) { count += v['count']; }); return {count: count}; }", verbose: false, out: "tweetresults", query: { verbose: true } } Mon Sep 26 19:24:59 [conn38] mr ns: jayed.tweets Mon Sep 26 19:24:59 [conn38] query jayed.system.js reslen:20 0ms Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tmp.mr.tweets_26_inc" } Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tmp.mr.tweets_26_inc Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tmp.mr.tweets_26_inc" } ntoreturn:1 reslen:62 0ms Mon Sep 26 19:24:59 [conn38] create collection jayed.tmp.mr.tweets_26_inc { autoIndexId: 0 } Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26_inc Mon Sep 26 19:24:59 [conn38] build index jayed.tmp.mr.tweets_26_inc { 0: 1 } Mon Sep 26 19:24:59 [conn38] external sort root: /var/lib/mongodb/_tmp/esort.1317079499.454318894/ Mon Sep 26 19:24:59 [conn38] external sort used : 0 files in 0 secs Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26_inc.$0_1 Mon Sep 26 19:24:59 [conn38] done building bottom layer, going to commit Mon Sep 26 19:24:59 [conn38] fastBuildIndex dupsToDrop:0 Mon Sep 26 19:24:59 [conn38] build index done 0 records 0 secs Mon Sep 26 19:24:59 [conn38] insert jayed.system.indexes 0ms Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tmp.mr.tweets_26" } Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tmp.mr.tweets_26 Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tmp.mr.tweets_26" } ntoreturn:1 reslen:62 0ms Mon Sep 26 19:24:59 [conn38] create collection jayed.tmp.mr.tweets_26 {} Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26 Mon Sep 26 19:24:59 [conn38] adding _id index for collection jayed.tmp.mr.tweets_26 Mon Sep 26 19:24:59 [conn38] build index jayed.tmp.mr.tweets_26 { _id: 1 } Mon Sep 26 19:24:59 [conn38] external sort root: /var/lib/mongodb/_tmp/esort.1317079499.1955500750/ Mon Sep 26 19:24:59 [conn38] external sort used : 0 files in 0 secs Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26.$ id Mon Sep 26 19:24:59 [conn38] done building bottom layer, going to commit Mon Sep 26 19:24:59 [conn38] fastBuildIndex dupsToDrop:0 Mon Sep 26 19:24:59 [conn38] build index done 0 records 0 secs Mon Sep 26 19:24:59 [conn38] query jayed.system.indexes nreturned:3 reslen:258 0ms Mon Sep 26 19:24:59 [conn38] build index jayed.tmp.mr.tweets_26 { count: -1.0 } Mon Sep 26 19:24:59 [conn38] external sort root: /var/lib/mongodb/_tmp/esort.1317079499.541709764/ Mon Sep 26 19:24:59 [conn38] external sort used : 0 files in 0 secs Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26.$count_-1 Mon Sep 26 19:24:59 [conn38] done building bottom layer, going to commit Mon Sep 26 19:24:59 [conn38] fastBuildIndex dupsToDrop:0 Mon Sep 26 19:24:59 [conn38] build index done 0 records 0 secs Mon Sep 26 19:24:59 [conn38] build index jayed.tmp.mr.tweets_26 { count: 1.0 } Mon Sep 26 19:24:59 [conn38] external sort root: /var/lib/mongodb/_tmp/esort.1317079499.1868679281/ Mon Sep 26 19:24:59 [conn38] external sort used : 0 files in 0 secs Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tmp.mr.tweets_26.$count_1 Mon Sep 26 19:24:59 [conn38] done building bottom layer, going to commit Mon Sep 26 19:24:59 [conn38] fastBuildIndex dupsToDrop:0 Mon Sep 26 19:24:59 [conn38] build index done 0 records 0 secs Mon Sep 26 19:24:59 [conn38] query jayed.system.indexes nreturned:1 reslen:98 0ms Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tweetresults" } Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tweetresults Mon Sep 26 19:24:59 [conn38] dropCollection: jayed.tweetresults Mon Sep 26 19:24:59 [conn38] dropIndexes done Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tweetresults" } ntoreturn:1 reslen:125 0ms Mon Sep 26 19:24:59 [conn38] run command admin.$cmd { renameCollection: "jayed.tmp.mr.tweets_26", to: "jayed.tweetresults" } Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tweetresults Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tweetresults.$ id Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tweetresults.$count_-1 Mon Sep 26 19:24:59 [conn38] New namespace: jayed.tweetresults.$count_1 Mon Sep 26 19:24:59 [conn38] command admin.$cmd command: { renameCollection: "jayed.tmp.mr.tweets_26", to: "jayed.tweetresults" } ntoreturn:1 reslen:37 1ms Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tmp.mr.tweets_26" } Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tmp.mr.tweets_26 Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tmp.mr.tweets_26" } ntoreturn:1 reslen:62 0ms Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tmp.mr.tweets_26" } Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tmp.mr.tweets_26 Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tmp.mr.tweets_26" } ntoreturn:1 reslen:62 0ms Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { drop: "tmp.mr.tweets_26_inc" } Mon Sep 26 19:24:59 [conn38] CMD: drop jayed.tmp.mr.tweets_26_inc Mon Sep 26 19:24:59 [conn38] dropCollection: jayed.tmp.mr.tweets_26_inc Mon Sep 26 19:24:59 [conn38] dropIndexes done Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { drop: "tmp.mr.tweets_26_inc" } ntoreturn:1 reslen:133 0ms Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { mapreduce: "tweets", map: "function (){ emit({ topic: this.topic }, { count: 1 }); }", reduce: "function(key, values) { var count = 0; values.forEach(function(v) { count += v['count']; } ); return {count: count} ; }", verbose: false, out: "tweetresults", query: { verbose: true } } ntoreturn:1 reslen:136 13ms Mon Sep 26 19:24:59 [conn38] insert jayed.system.indexes 0ms Mon Sep 26 19:24:59 [conn38] run command jayed.$cmd { getlasterror: 1, w: 1, wtimeout: 0 } Mon Sep 26 19:24:59 [conn38] command jayed.$cmd command: { getlasterror: 1, w: 1, wtimeout: 0 } ntoreturn:1 reslen:78 0ms
        Hide
        Brendan W. McAdams
        added a comment -

        Taking a look at this. Trying to sort out what the issue is. There are a few issues in mapping to the java library from Casbah in the current release which the fixes are queued up for 2.2

        Show
        Brendan W. McAdams
        added a comment - Taking a look at this. Trying to sort out what the issue is. There are a few issues in mapping to the java library from Casbah in the current release which the fixes are queued up for 2.2
        Hide
        Brendan W. McAdams
        added a comment -

        The JS Shell is probably doing some kind of implicit conversion for you which the raw JS engine isn't.

        Your map function is failing to compile. It is likely this:

        { this.topic }

        ,

        Which isn't a valid object. Either remove the braces or give it a value (

        { this.topic: 1 }

        ).

        Show
        Brendan W. McAdams
        added a comment - The JS Shell is probably doing some kind of implicit conversion for you which the raw JS engine isn't. Your map function is failing to compile. It is likely this: { this.topic } , Which isn't a valid object. Either remove the braces or give it a value ( { this.topic: 1 } ).
        Hide
        Jonathan Hendler
        added a comment -

        Thanks for the quick reply.

        Following this suggestion above I have:

        val map = "function (){ emit(

        { this.topic : 1 }

        ,

        { count: 1 }

        ); }"

        Which causes an explicit compile error [1]. In the comment above, I was able to have the js compile without error using

        val map = "function (){ emit(

        { topic: this.topic }

        ,

        { count: 1 }

        ); }"

        There was also no js compile error when using your suggestion

        val map = "function (){ emit( this.topic ,

        { count: 1 }

        ); }"

        But the problem is that no results are returned as much as a js compile error. I can only assume since this works from command line, that the error is not something I'm doing wrong, and that something gets lost between Casbah->Java->Mongo and back.

        If this is known issue the 2.2 release fixes, that seems straightforward. If the js shell and js raw is not identical, perhaps I'm missing some documentation about this?

        [1] -
        Tue Sep 27 08:59:26 [journal] lsn set 439830126
        Tue Sep 27 08:59:33 [initandlisten] connection accepted from 127.0.0.1:55486 #41
        Tue Sep 27 08:59:33 [conn41] run command admin.$cmd

        { isMaster: 1 }

        Tue Sep 27 08:59:33 [conn41] command admin.$cmd command:

        { isMaster: 1 }

        ntoreturn:1 reslen:71 0ms
        Tue Sep 27 08:59:33 [conn41] run command jayed.$cmd { mapreduce: "tweets", map: "function (){ emit(

        { this.topic : 1 }

        ,

        { count: 1 }

        ); }", reduce: "function(key, values) { var count = 0; values.forEach(function(v)

        { count += v['count']; }); return {count: count}; }", verbose: false, out: "tweetresults", query: { verbose: true } }
        Tue Sep 27 08:59:33 [conn41] mr ns: jayed.tweets
        Tue Sep 27 08:59:33 [conn41] query jayed.system.js reslen:20 0ms
        Tue Sep 27 08:59:33 [conn41] JS Error: SyntaxError: missing : after property id nofile_b:0
        Tue Sep 27 08:59:33 [conn41] compile failed for: function (){ emit({ this.topic : 1 }, { count: 1 }); }
        Tue Sep 27 08:59:33 [conn41] User Assertion: 13598:couldn't compile code for: _map
        Tue Sep 27 08:59:33 [conn41] mr failed, removing collection :: caused by :: 13598 couldn't compile code for: _map
        Tue Sep 27 08:59:33 [conn41] run command jayed.$cmd { drop: "tmp.mr.tweets_27" }
        Tue Sep 27 08:59:33 [conn41] CMD: drop jayed.tmp.mr.tweets_27
        Tue Sep 27 08:59:33 [conn41] command jayed.$cmd command: { drop: "tmp.mr.tweets_27" } ntoreturn:1 reslen:62 0ms
        Tue Sep 27 08:59:33 [conn41] run command jayed.$cmd { drop: "tmp.mr.tweets_27_inc" }
        Tue Sep 27 08:59:33 [conn41] CMD: drop jayed.tmp.mr.tweets_27_inc
        Tue Sep 27 08:59:33 [conn41] command jayed.$cmd command: { drop: "tmp.mr.tweets_27_inc" } ntoreturn:1 reslen:62 0ms
        Tue Sep 27 08:59:33 [conn41] command jayed.$cmd command: { mapreduce: "tweets", map: "function (){ emit({ this.topic : 1 }, { count: 1 }); }", reduce: "function(key, values) { var count = 0; values.forEach(function(v) { count += v['count']; }

        ); return

        {count: count}

        ; }", verbose: false, out: "tweetresults", query:

        { verbose: true }

        } ntoreturn:1 exception: couldn't compile code for: _map code:13598 reslen:136 9ms
        Tue Sep 27 08:59:33 [conn41] insert jayed.system.indexes 0ms
        Tue Sep 27 08:59:33 [conn41] run command jayed.$cmd

        { getlasterror: 1, w: 1, wtimeout: 0 }

        Tue Sep 27 08:59:33 [conn41] command jayed.$cmd command:

        { getlasterror: 1, w: 1, wtimeout: 0 }

        ntoreturn:1 reslen:78 0ms

        Show
        Jonathan Hendler
        added a comment - Thanks for the quick reply. Following this suggestion above I have: val map = "function (){ emit( { this.topic : 1 } , { count: 1 } ); }" Which causes an explicit compile error [1] . In the comment above, I was able to have the js compile without error using val map = "function (){ emit( { topic: this.topic } , { count: 1 } ); }" There was also no js compile error when using your suggestion val map = "function (){ emit( this.topic , { count: 1 } ); }" But the problem is that no results are returned as much as a js compile error. I can only assume since this works from command line, that the error is not something I'm doing wrong, and that something gets lost between Casbah->Java->Mongo and back. If this is known issue the 2.2 release fixes, that seems straightforward. If the js shell and js raw is not identical, perhaps I'm missing some documentation about this? [1] - Tue Sep 27 08:59:26 [journal] lsn set 439830126 Tue Sep 27 08:59:33 [initandlisten] connection accepted from 127.0.0.1:55486 #41 Tue Sep 27 08:59:33 [conn41] run command admin.$cmd { isMaster: 1 } Tue Sep 27 08:59:33 [conn41] command admin.$cmd command: { isMaster: 1 } ntoreturn:1 reslen:71 0ms Tue Sep 27 08:59:33 [conn41] run command jayed.$cmd { mapreduce: "tweets", map: "function (){ emit( { this.topic : 1 } , { count: 1 } ); }", reduce: "function(key, values) { var count = 0; values.forEach(function(v) { count += v['count']; }); return {count: count}; }", verbose: false, out: "tweetresults", query: { verbose: true } } Tue Sep 27 08:59:33 [conn41] mr ns: jayed.tweets Tue Sep 27 08:59:33 [conn41] query jayed.system.js reslen:20 0ms Tue Sep 27 08:59:33 [conn41] JS Error: SyntaxError: missing : after property id nofile_b:0 Tue Sep 27 08:59:33 [conn41] compile failed for: function (){ emit({ this.topic : 1 }, { count: 1 }); } Tue Sep 27 08:59:33 [conn41] User Assertion: 13598:couldn't compile code for: _map Tue Sep 27 08:59:33 [conn41] mr failed, removing collection :: caused by :: 13598 couldn't compile code for: _map Tue Sep 27 08:59:33 [conn41] run command jayed.$cmd { drop: "tmp.mr.tweets_27" } Tue Sep 27 08:59:33 [conn41] CMD: drop jayed.tmp.mr.tweets_27 Tue Sep 27 08:59:33 [conn41] command jayed.$cmd command: { drop: "tmp.mr.tweets_27" } ntoreturn:1 reslen:62 0ms Tue Sep 27 08:59:33 [conn41] run command jayed.$cmd { drop: "tmp.mr.tweets_27_inc" } Tue Sep 27 08:59:33 [conn41] CMD: drop jayed.tmp.mr.tweets_27_inc Tue Sep 27 08:59:33 [conn41] command jayed.$cmd command: { drop: "tmp.mr.tweets_27_inc" } ntoreturn:1 reslen:62 0ms Tue Sep 27 08:59:33 [conn41] command jayed.$cmd command: { mapreduce: "tweets", map: "function (){ emit({ this.topic : 1 }, { count: 1 }); }", reduce: "function(key, values) { var count = 0; values.forEach(function(v) { count += v['count']; } ); return {count: count} ; }", verbose: false, out: "tweetresults", query: { verbose: true } } ntoreturn:1 exception: couldn't compile code for: _map code:13598 reslen:136 9ms Tue Sep 27 08:59:33 [conn41] insert jayed.system.indexes 0ms Tue Sep 27 08:59:33 [conn41] run command jayed.$cmd { getlasterror: 1, w: 1, wtimeout: 0 } Tue Sep 27 08:59:33 [conn41] command jayed.$cmd command: { getlasterror: 1, w: 1, wtimeout: 0 } ntoreturn:1 reslen:78 0ms
        Hide
        Jonathan Hendler
        added a comment -

        I have a workaround working now, using eval()...

        db.eval( "map = function (){ emit(

        { topic: this.topic }

        ,

        { count: 1 }

        ); } ; ");
        db.eval( "reduce = function(key, values) { var count = 0; values.forEach(function(v)

        { count += v['count']; }

        ); return

        {count: count}

        ; }; ");
        db.eval( " db.tweets.mapReduce( map, reduce,

        { out: \"tweetresults\" }

        ); ");
        db.eval( " db.tweetresults.ensureIndex(

        {count : 1}

        ); ");

        Then I query the output table normally via casbah.

        Show
        Jonathan Hendler
        added a comment - I have a workaround working now, using eval()... db.eval( "map = function (){ emit( { topic: this.topic } , { count: 1 } ); } ; "); db.eval( "reduce = function(key, values) { var count = 0; values.forEach(function(v) { count += v['count']; } ); return {count: count} ; }; "); db.eval( " db.tweets.mapReduce( map, reduce, { out: \"tweetresults\" } ); "); db.eval( " db.tweetresults.ensureIndex( {count : 1} ); "); Then I query the output table normally via casbah.
        Hide
        Jonathan Hendler
        added a comment -

        This should provide some further clues.

        Using eval() - all queries work except for the find() listed below, which produces an undefined from eval(), but results from command line. db.getCollectionNames() lists the collection as existing.

        Code:

        val db = mongo("jayed")
        db.eval( "map = function (){ emit(

        { topic: this.topic }

        ,

        { count: 1 }

        ); } ; ");
        db.eval( "reduce = function(key, values) { var count = 0; values.forEach(function(v)

        { count += v['count']; }

        ); return

        {count: count}

        ; }; ");
        db.eval( " db.tweets.mapReduce( map, reduce,

        { out: \"tweetresults\" }

        ); ");
        ConfiggyObject.log.debug( db.eval( " db.getCollectionNames(); ").toString() );
        db.eval( " db.tweetresults.ensureIndex(

        {count : 1}

        ); ");
        ConfiggyObject.log.debug( db.eval( " db.getCollectionNames(); ").toString() );
        val json_string = db.eval( " db.tweetresults.find().sort(

        {count : -1}

        ); ").toString();
        ConfiggyObject.log.debug( db.eval( " db.getCollectionNames(); ").toString() );
        // val tweetresults = mongo("jayed")("tweetresults")
        // //tweetresults.ensureIndex( MongoDBObject("count" -> 1 ) , "topic_count_index", false )

        // val result_cursor = tweetresults.find().sort( MongoDBObject("count" -> 1 ) );
        // val result_map = result_cursor.toMap()
        // val json_string = com.mongodb.util.JSON.serialize( result_map )
        ConfiggyObject.log.debug("MR");
        ConfiggyObject.log.debug( json_string );

        MongoDB log

        Thu Sep 29 15:16:49 [conn72] command jayed.$cmd command: { $eval: " db.tweetresults.find().sort(

        {count : -1}

        ); ", args: {} } ntoreturn:1 reslen:102 0ms

        Custom log

        DEB [20110929-15:16:49.179] jayed: [ "system.indexes" , "tweetresults" , "tweets" , "user" , "users"]
        DEB [20110929-15:16:49.188] jayed: [ "system.indexes" , "tweetresults" , "tweets" , "user" , "users"]
        DEB [20110929-15:16:49.190] jayed: [ "system.indexes" , "tweetresults" , "tweets" , "user" , "users"]
        DEB [20110929-15:16:49.190] jayed: MR
        DEB [20110929-15:16:49.190] jayed:

        { "value" : "DBQuery: jayed.tweetresults -> undefined"}
        Show
        Jonathan Hendler
        added a comment - This should provide some further clues. Using eval() - all queries work except for the find() listed below, which produces an undefined from eval(), but results from command line. db.getCollectionNames() lists the collection as existing. Code: val db = mongo("jayed") db.eval( "map = function (){ emit( { topic: this.topic } , { count: 1 } ); } ; "); db.eval( "reduce = function(key, values) { var count = 0; values.forEach(function(v) { count += v['count']; } ); return {count: count} ; }; "); db.eval( " db.tweets.mapReduce( map, reduce, { out: \"tweetresults\" } ); "); ConfiggyObject.log.debug( db.eval( " db.getCollectionNames(); ").toString() ); db.eval( " db.tweetresults.ensureIndex( {count : 1} ); "); ConfiggyObject.log.debug( db.eval( " db.getCollectionNames(); ").toString() ); val json_string = db.eval( " db.tweetresults.find().sort( {count : -1} ); ").toString(); ConfiggyObject.log.debug( db.eval( " db.getCollectionNames(); ").toString() ); // val tweetresults = mongo("jayed")("tweetresults") // //tweetresults.ensureIndex( MongoDBObject("count" -> 1 ) , "topic_count_index", false ) // val result_cursor = tweetresults.find().sort( MongoDBObject("count" -> 1 ) ); // val result_map = result_cursor.toMap() // val json_string = com.mongodb.util.JSON.serialize( result_map ) ConfiggyObject.log.debug("MR"); ConfiggyObject.log.debug( json_string ); MongoDB log Thu Sep 29 15:16:49 [conn72] command jayed.$cmd command: { $eval: " db.tweetresults.find().sort( {count : -1} ); ", args: {} } ntoreturn:1 reslen:102 0ms Custom log DEB [20110929-15:16:49.179] jayed: [ "system.indexes" , "tweetresults" , "tweets" , "user" , "users"] DEB [20110929-15:16:49.188] jayed: [ "system.indexes" , "tweetresults" , "tweets" , "user" , "users"] DEB [20110929-15:16:49.190] jayed: [ "system.indexes" , "tweetresults" , "tweets" , "user" , "users"] DEB [20110929-15:16:49.190] jayed: MR DEB [20110929-15:16:49.190] jayed: { "value" : "DBQuery: jayed.tweetresults -> undefined"}
        Hide
        Jonathan Hendler
        added a comment -

        The behavior of eval(), the interactive shell, and other api's appears to not be consistant.

        This simple query creates the same error message as noted above - an undefined:
        val db = mongo("jayed")
        var find = "db.tweets.find(

        { topic : 'Apples' }

        )"
        ConfiggyObject.log.debug( db.eval( find ).toString())

        Custom Log:
        [20111001-09:29:06.775] jayed: { "tweets" : { "value" : "DBQuery: jayed.tweets -> undefined"}}

        Googling - found:

        http://stackoverflow.com/questions/3595265/basics-of-mongodb-scripts-how-to seemed to provide a clue about the undefined error above.

        The following query will print the results to the mongo log, one per row, but capturing output still gets an undefined!

        val db = mongo("jayed")
        db.eval( " printResult = function(r)

        { print(tojson(r))}" )
        var find = "db.tweets.find({ topic : 'Apples' })"
        find + ".forEach(printResult)"

        Mongo log:

        Sat Oct 1 09:45:33 [conn150] command jayed.$cmd command: { $eval: " printResult = function (r) { print(tojson(r))}

        ", args: {} } ntoreturn:1 reslen:88 0ms
        Sat Oct 1 09:45:33 [conn150] run command jayed.$cmd { $eval: "db.tweets.find(

        { topic : 'Apples' }

        )", args: {} }
        Sat Oct 1 09:45:33 [conn150] command jayed.$cmd command: { $eval: "db.tweets.find(

        { topic : 'Apples' }

        )", args: {} } ntoreturn:1 reslen:96 0ms
        Sat Oct 1 09:45:33 [conn150] run command jayed.$cmd { $eval: "db.tweets.find(

        { topic : 'Apples' }

        ).limit(20).forEach(printResult)", args: {} }
        Sat Oct 1 09:45:33 [conn150] query jayed.tweets ntoreturn:20 nreturned:1 reslen:255 0ms
        {
        "_id" : ObjectId("4e8687d1e4b00a1f35785dd1"),
        ...

        Show
        Jonathan Hendler
        added a comment - The behavior of eval(), the interactive shell, and other api's appears to not be consistant. This simple query creates the same error message as noted above - an undefined: val db = mongo("jayed") var find = "db.tweets.find( { topic : 'Apples' } )" ConfiggyObject.log.debug( db.eval( find ).toString()) Custom Log: [20111001-09:29:06.775] jayed: { "tweets" : { "value" : "DBQuery: jayed.tweets -> undefined"}} Googling - found: http://stackoverflow.com/questions/3595265/basics-of-mongodb-scripts-how-to seemed to provide a clue about the undefined error above. The following query will print the results to the mongo log, one per row, but capturing output still gets an undefined! val db = mongo("jayed") db.eval( " printResult = function(r) { print(tojson(r))}" ) var find = "db.tweets.find({ topic : 'Apples' })" find + ".forEach(printResult)" Mongo log: Sat Oct 1 09:45:33 [conn150] command jayed.$cmd command: { $eval: " printResult = function (r) { print(tojson(r))} ", args: {} } ntoreturn:1 reslen:88 0ms Sat Oct 1 09:45:33 [conn150] run command jayed.$cmd { $eval: "db.tweets.find( { topic : 'Apples' } )", args: {} } Sat Oct 1 09:45:33 [conn150] command jayed.$cmd command: { $eval: "db.tweets.find( { topic : 'Apples' } )", args: {} } ntoreturn:1 reslen:96 0ms Sat Oct 1 09:45:33 [conn150] run command jayed.$cmd { $eval: "db.tweets.find( { topic : 'Apples' } ).limit(20).forEach(printResult)", args: {} } Sat Oct 1 09:45:33 [conn150] query jayed.tweets ntoreturn:20 nreturned:1 reslen:255 0ms { "_id" : ObjectId("4e8687d1e4b00a1f35785dd1"), ...
        Hide
        Jonathan Hendler
        added a comment -

        Upgrades several libs in the shell. Same casbah version but with scala 2.9.1.

        +libraryDependencies += "com.mongodb.casbah" % "casbah_2.9.1" % "2.1.5-1"

        Show
        Jonathan Hendler
        added a comment - Upgrades several libs in the shell. Same casbah version but with scala 2.9.1. +libraryDependencies += "com.mongodb.casbah" % "casbah_2.9.1" % "2.1.5-1"
        Hide
        Jonathan Hendler
        added a comment -

        In the end, I've settled on using casbah's mapreduce, outputting to another table. There's no sharding yet, but group() prevents me from using sort(), limit() etc.

        This bug report is hopefully a helpful indication of the usability of querying. The eval() , command line, and other interfaces would be easier to learn with either better documentation and/or consistency.

        HTH

        Show
        Jonathan Hendler
        added a comment - In the end, I've settled on using casbah's mapreduce, outputting to another table. There's no sharding yet, but group() prevents me from using sort(), limit() etc. This bug report is hopefully a helpful indication of the usability of querying. The eval() , command line, and other interfaces would be easier to learn with either better documentation and/or consistency. HTH

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Days since reply:
              2 years, 29 weeks, 3 days ago
              Date of 1st Reply: