3

The procedure I follow to restart SmartTarget is as the following ;

  1. bin/Deployment-agent &
  2. bin/instance indexer start returns :kitchen already running for deployment

However, There is something causing Fredhopper services to stop or get killed . When we see that fredhopper is not working, then I check Fred instances with the following command;

"ps -ef | grep -i fred"

This only shows the deployment agent but nothing else even though I expect to see some fredhopper instances. In summary, fredhopper instances are gone couple of times a day.

Then I go to command line and issue the following line of command to run indexer again; ./bin/instance indexer start.

Then I check fred instances again; "ps -ef | grep -i fred".
I see the instances that I expect for one or two minutes. But after a few min they are gone again.

Status : [root@my fredhopper]# ./bin/instance indexer/ status

Exception in thread "Redirect: java.io.BufferedInputStream@29cb53c5 -> java.lang.UNIXProcess$ProcessPipeOutputStream@6040b6ef" java.lang.RuntimeException: java.io.IOException: Input/output error at com.fredhopper.launch.RedirectThread.run(RedirectThread.java:71) Caused by: java.io.IOException: Input/output error at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:272) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at com.fredhopper.launch.RedirectThread.run(RedirectThread.java:53)

Can you please make a wild guess why it could be happening ?

Where should I look at to see the exception?

And I really appricate if you can lead me the way I can investigate this. I checked all of the logs , but could not see a remarkable line of log that could help us understand the potential cause of the problem .

PS: The logs files I checked;
fredhopper.log , qserver.gc.log, qserver.log, searchindexer.log

Log Details (EDIT) Problem identified but has not been fixed yet ; Memory issues

kitchen.log

ERROR 03-11 13:53:36,244 - Start single instance - org.pentaho.di.core.exception.KettleException: 
Unexpected error occurred while launching entry [Is live node.0]
    at java.lang.Thread.run (Thread.java:744)
    at org.pentaho.di.job.entries.job.JobEntryJobRunner.run (JobEntryJobRunner.java:68)
    at org.pentaho.di.job.Job.execute (Job.java:562)
    at org.pentaho.di.job.Job.execute (Job.java:794)
    at org.pentaho.di.job.Job.execute (Job.java:650)
    at org.pentaho.di.job.entries.simpleeval.JobEntrySimpleEval.execute (JobEntrySimpleEval.java:525)

    at org.pentaho.di.job.Job.execute(Job.java:799)
    at org.pentaho.di.job.Job.execute(Job.java:562)
    at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:68)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
    at org.pentaho.di.job.entries.simpleeval.JobEntrySimpleEval.execute(JobEntrySimpleEval.java:525)
    at org.pentaho.di.job.Job.execute(Job.java:650)
    at org.pentaho.di.job.Job.execute(Job.java:794)
    ... 3 more

searchindexer.log

2015-11-03 14:16:44,085 PERFORMANCE [searchindexer] SearchIndexApp.java:373 40026 SearchIndexer total time= `448647' msec
2015-11-03 14:16:44,085 PROGRESS [searchindexer] SearchIndexApp.java:377 100019 Shutdown search indexer service.
2015-11-03 14:16:44,086 INFO  [Search-Indexer-Shutdown-Thread] SearchIndexApp.java:480 SearchIndexer stopped.

syncclient.log

2015-10-27 00:00:15,846 ERROR [SyncClientReplicationWorkerThread-1] ClientJob.java:216 50345 Error communicating to server `localhost':`10100'. Waiting `10' seconds before retrying... If you see this message rarely, the sync client will have recovered gracefully. || root@istsitportal:/home/fredhopper/fredhopper/data/installations/FAS tags/fas/fas-7.5.0.16/engine@107262
java.io.IOException: Not enough disk space in /home/fredhopper/fredhopper/data/instances/query/data/indices/itemstore (Needed 104857600, available 0)
    at com.fredhopper.replication.client.ClientReplicationJob.receiveReplicationItems(ClientReplicationJob.java:820)
    at com.fredhopper.replication.client.ClientReplicationJob.doExecute(ClientReplicationJob.java:270)
    at com.fredhopper.replication.client.ClientJob.execute(ClientJob.java:179)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:534

Details;
SmartTarget 2014, Product Version: 2.0.0.39 ;

java version "1.7.0_51"

  • In which log do you see that error? Also, can you do a ps -ef | grep java instead? it maybe that there are processes running without "fred" in it.. – Raimond Nov 3 '15 at 11:48
  • Raimond, the error is seen right after I issue the "status" command for the indexer. Yes, there are are some processes running, and I am aware of it. But Java instances I think should not be related to this problem. – shamaleyte Nov 9 '15 at 15:34
  • More a note than an answer: the IOException regarding the inputstream might be a red herring, and caused by an older Java version, and or fixed by a newer deployment agent - I vaguely remember that it was caused by an ugly race in the Java process classes that was hard to fix properly. What exact version of Fredhopper, SmartTarget, deployment-agent, and Java are you using here? – ankon Nov 10 '15 at 16:52
  • @ankon; SmartTarget 2014, Product Version: 2.0.0.39 ; java version "1.7.0_51" – shamaleyte Nov 12 '15 at 11:32
3

"kitchen already running for deployment" suggests that there is already an instance running -- or at least that Fredhopper thinks there is.

Try to first make sure the instance is stopped (i.e. bin\instance indexer stop). Verify that there aren't any Java processes running from that instance.

After that is done, check if you have any .socket files in the tmp folder of the instance. If you do, it suggests that someone shut down the instance the wrong way so it didn't get a chance to clean up. You can then try deleting those .socket files and start the instance again.

  • Peter I understand that it is expected for status to be "kitchen already running" right after I start the indexer. But the weird thing is that it suddenly stops after a while and gives "IO error" with the status command . I checked the following path and could not find any .socket files . Am I looking at the right path ? /home/fredhopper/fredhopper/data/instances/indexer/tmp – shamaleyte Nov 4 '15 at 14:15
  • Why would you say that error is "expected"? In my opinion, the two issues are related. Yes, that is the right path for the socket files - assuming you are on the latest supported Fredhopper version. In previous versions I believe it places them in the logs folder and they were named something slightly different. – Peter Kjaer Nov 4 '15 at 15:24
  • I said "expected" because I am checking the status right after starting it. Am I wrong about this? but anyways, I edited the problem description and there is a "disk space" problem. – shamaleyte Nov 10 '15 at 16:01
2

Some pointers to look at:

  1. the deployment-agent.log should contain more information about which processes were started, and possibly why they failed.
  2. "kitchen already running for deployment" is a round-about way of pointing to the log file: the kitchen.log in the deployment instance, i.e. in your case that would be in /home/fredhopper/fredhopper/data/instances/deployment/log/kitchen.log.
  • edited the description with log details. – shamaleyte Nov 10 '15 at 15:59
2

The answer is found in syncclient.log:

java.io.IOException: Not enough disk space in /home/fredhopper/fredhopper/data/instances/query/data/indices/itemstore (Needed 104857600, available 0)

I guess you first need to increase that before starting Fredhopper.

0

Addition to the suggestions given , I would like to state that my problem is definitely related to the memory.

This link may help increasing the memory allocation of instances.
https://www.fredhopper.com/learningcenter/display/learningcenter/Change+Qserver+Memory+Setting

I increased instance3_searchindexer_memory from 900 to 4000. For the time being, I do not seem to have a similar problem but let's see, it is just the second day of this solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.