Spark Mongo 笔记

spark mongo pipeline partitioning failed

ClassNotFoundException: ...MongoDefaultPartition

Partitioning using the 'DefaultMongoPartitioner$' failed.
java.lang.NoSuchMethodError: com.mongodb.client.MongoCollection.countDocuments(Lorg/bson/conversions/Bson;)

目前将partitioner设置为MongoSplitVectorPartitioner可解决

SparkConf().set("spark.mongodb.input.partitioner", "MongoSplitVectorPartitioner")

参考
https://docs.mongodb.com/spark-connector/current/configuration/index.html#input-configuration

no permission on database config

new SparkConf().set("spark.mongodb.input.uri", "mongodb://user:pwd@host:port/db.collection")

需要同时设置mongodb user的 config db read 权限

db.updateUser("readonly",{"roles":[{"role":"read","db":"yourdb"},{"role":"read","db":"config"}]});