# Fredhopper instances get killed

The procedure I follow to restart SmartTarget is as the following ;

1. bin/Deployment-agent &
2. bin/instance indexer start returns :kitchen already running for deployment

However, There is something causing Fredhopper services to stop or get killed . When we see that fredhopper is not working, then I check Fred instances with the following command;

"ps -ef | grep -i fred"

This only shows the deployment agent but nothing else even though I expect to see some fredhopper instances. In summary, fredhopper instances are gone couple of times a day.

Then I go to command line and issue the following line of command to run indexer again; ./bin/instance indexer start.

Then I check fred instances again; "ps -ef | grep -i fred".
I see the instances that I expect for one or two minutes. But after a few min they are gone again.

Status : [root@my fredhopper]# ./bin/instance indexer/ status

Exception in thread "Redirect: java.io.BufferedInputStream@29cb53c5 -> java.lang.UNIXProcess$ProcessPipeOutputStream@6040b6ef" java.lang.RuntimeException: java.io.IOException: Input/output error at com.fredhopper.launch.RedirectThread.run(RedirectThread.java:71) Caused by: java.io.IOException: Input/output error at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:272) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at com.fredhopper.launch.RedirectThread.run(RedirectThread.java:53) Can you please make a wild guess why it could be happening ? Where should I look at to see the exception? And I really appricate if you can lead me the way I can investigate this. I checked all of the logs , but could not see a remarkable line of log that could help us understand the potential cause of the problem . PS: The logs files I checked; fredhopper.log , qserver.gc.log, qserver.log, searchindexer.log ## Log Details (EDIT) Problem identified but has not been fixed yet ; Memory issues kitchen.log ERROR 03-11 13:53:36,244 - Start single instance - org.pentaho.di.core.exception.KettleException: Unexpected error occurred while launching entry [Is live node.0] at java.lang.Thread.run (Thread.java:744) at org.pentaho.di.job.entries.job.JobEntryJobRunner.run (JobEntryJobRunner.java:68) at org.pentaho.di.job.Job.execute (Job.java:562) at org.pentaho.di.job.Job.execute (Job.java:794) at org.pentaho.di.job.Job.execute (Job.java:650) at org.pentaho.di.job.entries.simpleeval.JobEntrySimpleEval.execute (JobEntrySimpleEval.java:525) at org.pentaho.di.job.Job.execute(Job.java:799) at org.pentaho.di.job.Job.execute(Job.java:562) at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:68) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.NullPointerException at org.pentaho.di.job.entries.simpleeval.JobEntrySimpleEval.execute(JobEntrySimpleEval.java:525) at org.pentaho.di.job.Job.execute(Job.java:650) at org.pentaho.di.job.Job.execute(Job.java:794) ... 3 more  searchindexer.log 2015-11-03 14:16:44,085 PERFORMANCE [searchindexer] SearchIndexApp.java:373 40026 SearchIndexer total time= 448647' msec 2015-11-03 14:16:44,085 PROGRESS [searchindexer] SearchIndexApp.java:377 100019 Shutdown search indexer service. 2015-11-03 14:16:44,086 INFO [Search-Indexer-Shutdown-Thread] SearchIndexApp.java:480 SearchIndexer stopped.  syncclient.log 2015-10-27 00:00:15,846 ERROR [SyncClientReplicationWorkerThread-1] ClientJob.java:216 50345 Error communicating to server localhost':10100'. Waiting 10' seconds before retrying... If you see this message rarely, the sync client will have recovered gracefully. || root@istsitportal:/home/fredhopper/fredhopper/data/installations/FAS tags/fas/fas-7.5.0.16/engine@107262 java.io.IOException: Not enough disk space in /home/fredhopper/fredhopper/data/instances/query/data/indices/itemstore (Needed 104857600, available 0) at com.fredhopper.replication.client.ClientReplicationJob.receiveReplicationItems(ClientReplicationJob.java:820) at com.fredhopper.replication.client.ClientReplicationJob.doExecute(ClientReplicationJob.java:270) at com.fredhopper.replication.client.ClientJob.execute(ClientJob.java:179) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:534


Details;
SmartTarget 2014, Product Version: 2.0.0.39 ;

java version "1.7.0_51"

• In which log do you see that error? Also, can you do a ps -ef | grep java instead? it maybe that there are processes running without "fred" in it.. Nov 3, 2015 at 11:48
• Raimond, the error is seen right after I issue the "status" command for the indexer. Yes, there are are some processes running, and I am aware of it. But Java instances I think should not be related to this problem. Nov 9, 2015 at 15:34
• More a note than an answer: the IOException regarding the inputstream might be a red herring, and caused by an older Java version, and or fixed by a newer deployment agent - I vaguely remember that it was caused by an ugly race in the Java process classes that was hard to fix properly. What exact version of Fredhopper, SmartTarget, deployment-agent, and Java are you using here? Nov 10, 2015 at 16:52
• @ankon; SmartTarget 2014, Product Version: 2.0.0.39 ; java version "1.7.0_51" Nov 12, 2015 at 11:32

"kitchen already running for deployment" suggests that there is already an instance running -- or at least that Fredhopper thinks there is.

Try to first make sure the instance is stopped (i.e. bin\instance indexer stop). Verify that there aren't any Java processes running from that instance.

After that is done, check if you have any .socket files in the tmp folder of the instance. If you do, it suggests that someone shut down the instance the wrong way so it didn't get a chance to clean up. You can then try deleting those .socket files and start the instance again.

• Peter I understand that it is expected for status to be "kitchen already running" right after I start the indexer. But the weird thing is that it suddenly stops after a while and gives "IO error" with the status command . I checked the following path and could not find any .socket files . Am I looking at the right path ? /home/fredhopper/fredhopper/data/instances/indexer/tmp Nov 4, 2015 at 14:15
• Why would you say that error is "expected"? In my opinion, the two issues are related. Yes, that is the right path for the socket files - assuming you are on the latest supported Fredhopper version. In previous versions I believe it places them in the logs folder and they were named something slightly different. Nov 4, 2015 at 15:24
• I said "expected" because I am checking the status right after starting it. Am I wrong about this? but anyways, I edited the problem description and there is a "disk space" problem. Nov 10, 2015 at 16:01

Some pointers to look at:

1. the deployment-agent.log should contain more information about which processes were started, and possibly why they failed.
2. "kitchen already running for deployment" is a round-about way of pointing to the log file: the kitchen.log in the deployment instance, i.e. in your case that would be in /home/fredhopper/fredhopper/data/instances/deployment/log/kitchen.log.
• edited the description with log details. Nov 10, 2015 at 15:59

The answer is found in syncclient.log:

java.io.IOException: Not enough disk space in /home/fredhopper/fredhopper/data/instances/query/data/indices/itemstore (Needed 104857600, available 0)

I guess you first need to increase that before starting Fredhopper.

Addition to the suggestions given , I would like to state that my problem is definitely related to the memory.

This link may help increasing the memory allocation of instances.
https://www.fredhopper.com/learningcenter/display/learningcenter/Change+Qserver+Memory+Setting

I increased instance3_searchindexer_memory from 900 to 4000. For the time being, I do not seem to have a similar problem but let's see, it is just the second day of this solution.