SolrCloud: unable to create collection, blocking issues
I am trying to implement SolrCloud and everything works fine until I try to create a collection with 6 shards. My setup is like this:
- 5 virtual servers, all running Ubuntu 14.04, hosted by the same company in different data centers.
- 3 servers running ZooKeeper 3.4.6 for ensemble
- 2 servers each running Solr 5.1.0 server (Jetty)
- Each Solr instance has a 2TB, ext4 secondary disk for indexes installed in / solrData / Indexes. I set this value in
solrconfig.xml
via<dataDir>/solrData/Indexes</dataDir>
and loaded it into the ZooKeeper ensemble. Please note that these secondary drives are neither NAS nor NFS, which I know can cause problems.solr
Owned by the user/solrData
.
All intra-server communication is done over a private IP as they are all hosted by the same company. I am using iptables for firewall and ports are open and all servers are communicating successfully. The Config upload to ZooKeeper was successful and I can see through the Solr administration interface that both nodes are available.
The problem arises when I try to create a collection using the following command:
http://xxx.xxx.xxx.xxx:8983/solr/admin/collections?action=CREATE&name=coll1&maxShardsPerNode=6&router.name=implicit&shards=shard1,shard2,shard3,shard4,shard5,shard6&router.field=shard&async=4444
Thanks to the Solr UI protocol, I can see several index creation commands being issued at the same time:
6/25/2015, 7:55:45 AM WARN SolrCore [coll1_shard2_replica1] Solr index directory '/solrData/Indexes/index' doesn't exist. Creating new index...
6/25/2015, 7:55:45 AM WARN SolrCore [coll1_shard1_replica2] Solr index directory '/solrData/Indexes/index' doesn't exist. Creating new index...
Ultimately the task gets the message as complete, but I have blocking errors in the log:
Error creating core [coll1_shard2_replica1]: Lock obtain timed out: SimpleFSLock@/solrData/Indexes/index/write.lock
SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
Error closing IndexWriter
If I look at the cloud graph, maybe a couple of shards will be created, others will be closed or restored, and if I restart Solr, none of the cores can start.
Now, I know what you are going to say: follow this SO post and change solrconfig.xml
the lock options to this:
<unlockOnStartup>true</unlockOnStartup>
<lockType>simple</lockType>
I did it and it had no effect. Hence the question. I'm about to release one Solr to production, which I don't like to do. Does anyone know how to fix this?
source to share
Based on the log entry entered, it looks like Solr can create a data directory (index) for EVERY shard in the same folder.
Solr index directory '/solrData/Indexes/index' doesn't exist. Creating new index...
This post has been shown for two different collections and it links to the same location. What I usually do is change my Solr Home to a different directory where all the materials of the collection instance will be created. Then I manually change the core.properties properties for each shard to indicate the location of the index data.
source to share