If you are in Big Data Service 3.0.9+, both Spark and Hive use the same catalog 'hive' and ACID in Hive is disabled by default. To enable ACID, the following configs must be updated using Ambari UI. After enabling ACID in Hive, Spark can't read/write in Hive managed tables. Therefore, be sure to update the catalog within Spark. Only the external table works correctly from Spark.
-
Access Apache Ambari.
-
From the side toolbar, under Services click Hive.
-
Click Configs.
-
Under Custom hive-site, enter the following configuration information:
- hive.support.concurrency: True
- hive.txn.manager: org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
- hive.enforce.bucketing: True
- hive.exec.dynamic.partition.mode: Nonstrict
- hive.compactor.initiator.on: True
- hive.compactor.worker.threads: 5
- hive.strict.managed.tables: True
- hive.create.as.insert.only: True
- metastore.create.as.acid: True
-
Click Add Property.
-
Restart the Hive service.