Publishing fails with SI4T and aws-cloud-search AmazonWebServiceClient.beforeMarshalling - NoSuchMethodError

Question

We have migrated SDL Tridion 2013 SP1 to Web 8.5 and while configuring AWS cloud search by taking jars from following git hub repository https://github.com/SI4T/aws-cloud-search

After configuring AWS cloud search as per documentation when we published, It's failing at Deployment Commit phase. When looked through logs found below stack trace with error

AmazonWebServiceClient.beforeMarshalling - NoSuchMethodError

2018-12-08 06:29:54,016 ERROR TridionCommitStep - Commit failed for execution id:tcm:0-2414563-66560
com.sdl.delivery.deployer.api.processing.exception.ProcessingException: Unable to commit transactions. ExecutionId:tcm:0-2414563-66560
    at com.sdl.delivery.deployer.steps.TridionTransactionalExecutableStep.commitTransaction(TridionTransactionalExecutableStep.java:78)
    at com.sdl.delivery.deployer.steps.TridionCommitStep.process(TridionCommitStep.java:28)
    at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor.execute(StepActor.scala:72)
    at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor.com$sdl$delivery$deployer$engine$pipeline$actor$StepActor$$processStep(StepActor.scala:57)
    at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor$$anonfun$receive$1.applyOrElse(StepActor.scala:40)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:480)
    at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor.aroundReceive(StepActor.scala:26)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: com.tridion.broker.StorageException: Commit failed for transaction tcm:0-2414563-66560 because of com.amazonaws.AmazonWebServiceClient.beforeMarshalling(Lcom/amazonaws/AmazonWebServiceRequest;)Lcom/amazonaws/AmazonWebServiceRequest;
    at com.tridion.storage.management.TransactionManagerImpl.commitTransaction(TransactionManagerImpl.java:70)
    at com.tridion.storage.StorageManagerFactory.commitTransaction(StorageManagerFactory.java:226)
    at com.sdl.delivery.deployer.steps.TridionTransactionalExecutableStep.commitTransaction(TridionTransactionalExecutableStep.java:76)
    ... 13 common frames omitted
Caused by: java.lang.NoSuchMethodError: com.amazonaws.AmazonWebServiceClient.beforeMarshalling(Lcom/amazonaws/AmazonWebServiceRequest;)Lcom/amazonaws/AmazonWebServiceRequest;
    at com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient.uploadDocuments(AmazonCloudSearchDomainClient.java:299)
    at org.si4t.cloudsearch.CloudSearchIndexDispatcher.addDocuments(CloudSearchIndexDispatcher.java:101)
    at org.si4t.cloudsearch.CloudSearchIndexer.dispatchAddContentToCloudSearch(CloudSearchIndexer.java:298)
    at org.si4t.cloudsearch.CloudSearchIndexer.commitAddContentToCloudSearch(CloudSearchIndexer.java:221)
    at org.si4t.cloudsearch.CloudSearchIndexer.commit(CloudSearchIndexer.java:140)
    at com.tridion.storage.si4t.SearchIndexProcessor.triggerIndexing(SearchIndexProcessor.java:274)
    at com.tridion.storage.si4t.JPASearchDAOFactory.commitTransaction(JPASearchDAOFactory.java:189)
    at com.tridion.storage.management.TransactionManagerImpl.commitTransaction(TransactionManagerImpl.java:65)
    ... 15 common frames omitted

posting cd storage config and search DAO bundle configued

<Storages>
            <StorageBindings>
                <Bundle src="SearchDAOBundle.xml"/>
            </StorageBindings>
            <Storage Class="com.tridion.storage.si4t.JPASearchDAOFactory" Id="sqlServer" Type="persistence" dialect="MSSQL">
                <Pool CheckoutTimeout="120" IdleTimeout="120" MonitorInterval="60" Size="5" Type="jdbc"/>
                 <DataSource Class="com.microsoft.sqlserver.jdbc.SQLServerDataSource">
                    <Property Name="serverName" Value="DEV02\SQL01"/>
                    <Property Name="portNumber" Value="9561"/>
                    <Property Name="databaseName" Value="Tridion_Broker_Live"/>
                    <Property Name="user" Value="TridionBrokerUser"/>
                    <Property Name="password" Value="Tridion"/>
                </DataSource>
                <Indexer 
            Class="org.si4t.cloudsearch.CloudSearchIndexer" 
            DocExtensions="pdf,docx,doc,xls,xlsx,pptx,ppt" 
            access_key_id="AKIAIKEW" 
            authentication="explicit" 
            documentEndpoint="doc-dev-le-5yist4ubp3pr2gwevm22uq.eu-west-1.cloudsearch.amazonaws.com"
            indexBatchSize="20" 
            secret_access_key="QT54AX8PC5TZNOqM9/pS"/>      
            </Indexer>
            </Storage>
            <Storage Class="com.tridion.storage.persistence.JPADAOFactory" Id="defaultdb" Type="persistence" dialect="MSSQL">
                <Pool CheckoutTimeout="120" IdleTimeout="120" MonitorInterval="60" Size="5" Type="jdbc"/>
                <DataSource Class="com.microsoft.sqlserver.jdbc.SQLServerDataSource">
                    <Property Name="serverName" Value="DEV02\SQL01"/>
                    <Property Name="portNumber" Value="9561"/>
                    <Property Name="databaseName" Value="Tridion_Broker_Live"/>
                    <Property Name="user" Value="TridionBrokerUser"/>
                    <Property Name="password" Value="Tridion"/>
                </DataSource>
            </Storage>
            <Storage Class="com.tridion.storage.filesystem.FSDAOFactory" Id="ce_index_files" Type="filesystem" defaultFilesystem="false" defaultStorage="false">
                <Root Path="D:\DevLivePublic\live\website\SearchIndex\Resources"/>           
            </Storage>
        </Storages>

Search DAO Bundle

<StorageDAOBundle type="persistence">
        <StorageDAO typeMapping="Page" class="com.tridion.storage.si4t.dao.JPASearchPageDAO"/>
        <StorageDAO typeMapping="Binary" class="com.tridion.storage.si4t.dao.JPASearchBinaryContentDAO"/>
        <StorageDAO typeMapping="ComponentPresentation" class="com.tridion.storage.si4t.dao.JPASearchComponentPresentationDAO"/>
    </StorageDAOBundle>

can you please help

Can you please edit your question and share your pom.xml file and list of dependencies jars file names which you have prepared and uploaded to deployer service? — Velmurugan, Dec 17, 2018 at 22:05

Velmurugan · Accepted Answer · 2018-12-21 13:10:14Z

I think I had similar issue,

The current version of aws-cloud-search source pom.xml used with following versions

<tridion.version>7.1.0</tridion.version>
<si4t.version>1.0</si4t.version>
<amazonaws.version>1.10.16</amazonaws.version>

Note:

SDL Tridion Sites 8.5 (formerly SDL Web) - deployer service used following jars for AWS SQS.

aws-java-sdk-core-1.9.6.jar
aws-java-sdk-sqs-1.9.6.jar

I believe your current aws-cloud-search build has AWS SDK 1.10.16, which you have to reduce to use 1.9.6 because it's conflict with 1.10.16.

Also noticed that current aws-cloud-search source pom.xml used artifactId aws-java-sdk, which is downloading all the AWS SDK services jar files which are not necessary, instead use artifactId aws-java-sdk-cloudsearch to download only cloudsearch jar only.

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-cloudsearch</artifactId>
    <version>1.9.6</version>
</dependency>

You can try using this pom.xml and rebuild and try to test it again.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.si4t</groupId>
    <artifactId>si4t-cloudsearch</artifactId>
    <version>1.0-SNAPSHOT</version>

    <packaging>jar</packaging>

    <properties>
        <tridion.version>8.5.0</tridion.version>
        <si4t.version>1.3</si4t.version>
        <amazonaws.version>1.9.6</amazonaws.version>
        <commons-io.version>2.4</commons-io.version>
        <httpclient.version>4.4</httpclient.version>
        <gson.version>2.2.2</gson.version>
        <slf4j-api.version>1.6.4</slf4j-api.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>2.3.1</version>
                <configuration>
                    <archive>
                        <addMavenDescriptor>false</addMavenDescriptor>
                    </archive>
                    <outputDirectory>${project.basedir}/dist</outputDirectory>
                    <!-- TODO: update this when using proper MVN versioning -->
                    <finalName>si4t-cloudsearch-1.0</finalName>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-dependency-plugin</artifactId>
                <executions>
                    <!-- Do note the exclusions. These should come with the Tridion Deployer-->
                    <!-- TODO: check whether we need all of the AWS jars. If not - exclude! -->
                    <execution>
                        <id>install</id>
                        <phase>install</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.basedir}/lib</outputDirectory>
                            <excludeGroupIds>com.tridion,org.slf4j,org.apache,commons-io,commons-codec,commons-logging</excludeGroupIds>
                        </configuration>
                    </execution>
                    <execution>
                        <id>package</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.basedir}/lib</outputDirectory>
                            <excludeGroupIds>com.tridion,org.slf4j,org.apache,commons-io,commons-codec,commons-logging</excludeGroupIds>
                        </configuration>
                    </execution>
                    <execution>
                        <id>compile</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.basedir}/lib</outputDirectory>
                            <excludeGroupIds>com.tridion,org.slf4j,org.apache,commons-io,commons-codec,commons-logging</excludeGroupIds>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <!-- TODO: upload si4t to Maven -->
            <groupId>org.si4t</groupId>
            <artifactId>si4t-se</artifactId>
            <version>${si4t.version}</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-cloudsearch</artifactId>
            <version>${amazonaws.version}</version>
        </dependency>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>${commons-io.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>${httpclient.version}</version>
        </dependency>
        <dependency>
            <groupId>com.tridion</groupId>
            <artifactId>cd_model</artifactId>
            <version>${tridion.version}</version>
        </dependency>
        <dependency>
            <groupId>com.tridion</groupId>
            <artifactId>cd_core</artifactId>
            <version>${tridion.version}</version>
        </dependency>
        <!-- Web 8 new -->

        <dependency>
            <groupId>com.tridion</groupId>
            <artifactId>cd_common_util</artifactId>
            <version>${tridion.version}</version>
        </dependency>

        <dependency>
            <groupId>com.tridion</groupId>
            <artifactId>cd_common_config_legacy</artifactId>
            <version>${tridion.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>${slf4j-api.version}</version>
        </dependency>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>${gson.version}</version>
        </dependency>
    </dependencies>
</project>

Updated:

SDL.DXA.Search.Module.2.0\modules\Search\deployer\lib\si4t-cloudsearch - which is using the correct version of AWS (1.9.6), you could also use that there also I can see it has all AWS SDK services jars. it's not the problem, I would say it's not necessary to keep all the AWS SDK files other than cloudsearch.

I hope it helps and it will fix your issue.

Thank you velmurgan, I will try your solution and update my findings. one more doubt, can’t we use AWS cloud jar files from DXA 2.0 search module instead of rebuilding the jars? — Prasanna K Gollamudi, Dec 19, 2018 at 1:49
SDL.DXA.Search.Module.2.0\modules\Search\deployer\lib\si4t-cloudsearch - which is using the correct version of AWS (1.9.6), you could also use that there also I can see it has all AWS SDK services jars. it's not the problem, I would say it's not necessary to keep all the AWS SDK files other than cloudsearch. — Velmurugan, Dec 19, 2018 at 7:43
Can you please add above comment as answer so that i can accept it. — Prasanna K Gollamudi, Dec 21, 2018 at 12:58

Stack Exchange Network

Publishing fails with SI4T and aws-cloud-search AmazonWebServiceClient.beforeMarshalling - NoSuchMethodError

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged
publishing
web8.5
si4t
publishingfailed
or ask your own question.

Hot Network Questions

Publishing fails with SI4T and aws-cloud-search AmazonWebServiceClient.beforeMarshalling - NoSuchMethodError

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged publishingweb8.5si4tpublishingfailed or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
publishing
web8.5
si4t
publishingfailed
or ask your own question.