Fredhopper instances get killed

Question

The procedure I follow to restart SmartTarget is as the following ;

1. bin/Deployment-agent &
2. bin/instance indexer start returns :kitchen already running for deployment

However, There is something causing Fredhopper services to stop or get killed . When we see that fredhopper is not working, then I check Fred instances with the following command;

"ps -ef | grep -i fred"

This only shows the deployment agent but nothing else even though I expect to see some fredhopper instances. In summary, fredhopper instances are gone couple of times a day.

Then I go to command line and issue the following line of command to run indexer again; ./bin/instance indexer start.

Then I check fred instances again; "ps -ef | grep -i fred".
I see the instances that I expect for one or two minutes. But after a few min they are gone again.

Status : [root@my fredhopper]# ./bin/instance indexer/ status

Exception in thread "Redirect: java.io.BufferedInputStream@29cb53c5 -> java.lang.UNIXProcess$ProcessPipeOutputStream@6040b6ef" java.lang.RuntimeException: java.io.IOException: Input/output error at com.fredhopper.launch.RedirectThread.run(RedirectThread.java:71) Caused by: java.io.IOException: Input/output error at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:272) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at com.fredhopper.launch.RedirectThread.run(RedirectThread.java:53)

Can you please make a wild guess why it could be happening ?

Where should I look at to see the exception?

And I really appricate if you can lead me the way I can investigate this. I checked all of the logs , but could not see a remarkable line of log that could help us understand the potential cause of the problem .

PS: The logs files I checked;
fredhopper.log , qserver.gc.log, qserver.log, searchindexer.log

Log Details (EDIT) Problem identified but has not been fixed yet ; Memory issues

kitchen.log

ERROR 03-11 13:53:36,244 - Start single instance - org.pentaho.di.core.exception.KettleException: 
Unexpected error occurred while launching entry [Is live node.0]
    at java.lang.Thread.run (Thread.java:744)
    at org.pentaho.di.job.entries.job.JobEntryJobRunner.run (JobEntryJobRunner.java:68)
    at org.pentaho.di.job.Job.execute (Job.java:562)
    at org.pentaho.di.job.Job.execute (Job.java:794)
    at org.pentaho.di.job.Job.execute (Job.java:650)
    at org.pentaho.di.job.entries.simpleeval.JobEntrySimpleEval.execute (JobEntrySimpleEval.java:525)

    at org.pentaho.di.job.Job.execute(Job.java:799)
    at org.pentaho.di.job.Job.execute(Job.java:562)
    at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:68)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
    at org.pentaho.di.job.entries.simpleeval.JobEntrySimpleEval.execute(JobEntrySimpleEval.java:525)
    at org.pentaho.di.job.Job.execute(Job.java:650)
    at org.pentaho.di.job.Job.execute(Job.java:794)
    ... 3 more

searchindexer.log

2015-11-03 14:16:44,085 PERFORMANCE [searchindexer] SearchIndexApp.java:373 40026 SearchIndexer total time= `448647' msec
2015-11-03 14:16:44,085 PROGRESS [searchindexer] SearchIndexApp.java:377 100019 Shutdown search indexer service.
2015-11-03 14:16:44,086 INFO  [Search-Indexer-Shutdown-Thread] SearchIndexApp.java:480 SearchIndexer stopped.

syncclient.log

2015-10-27 00:00:15,846 ERROR [SyncClientReplicationWorkerThread-1] ClientJob.java:216 50345 Error communicating to server `localhost':`10100'. Waiting `10' seconds before retrying... If you see this message rarely, the sync client will have recovered gracefully. || root@istsitportal:/home/fredhopper/fredhopper/data/installations/FAS tags/fas/fas-7.5.0.16/engine@107262
java.io.IOException: Not enough disk space in /home/fredhopper/fredhopper/data/instances/query/data/indices/itemstore (Needed 104857600, available 0)
    at com.fredhopper.replication.client.ClientReplicationJob.receiveReplicationItems(ClientReplicationJob.java:820)
    at com.fredhopper.replication.client.ClientReplicationJob.doExecute(ClientReplicationJob.java:270)
    at com.fredhopper.replication.client.ClientJob.execute(ClientJob.java:179)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:534

Details;
SmartTarget 2014, Product Version: 2.0.0.39 ;

java version "1.7.0_51"

Answer 1

"kitchen already running for deployment" suggests that there is already an instance running -- or at least that Fredhopper thinks there is.

Try to first make sure the instance is stopped (i.e. bin\instance indexer stop). Verify that there aren't any Java processes running from that instance.

After that is done, check if you have any .socket files in the tmp folder of the instance. If you do, it suggests that someone shut down the instance the wrong way so it didn't get a chance to clean up. You can then try deleting those .socket files and start the instance again.

Answer 2

Some pointers to look at:

1. the deployment-agent.log should contain more information about which processes were started, and possibly why they failed.
2. "kitchen already running for deployment" is a round-about way of pointing to the log file: the kitchen.log in the deployment instance, i.e. in your case that would be in /home/fredhopper/fredhopper/data/instances/deployment/log/kitchen.log.

Answer 3

The answer is found in syncclient.log:

java.io.IOException: Not enough disk space in /home/fredhopper/fredhopper/data/instances/query/data/indices/itemstore (Needed 104857600, available 0)

I guess you first need to increase that before starting Fredhopper.

Answer 4

Addition to the suggestions given , I would like to state that my problem is definitely related to the memory.

This link may help increasing the memory allocation of instances.
https://www.fredhopper.com/learningcenter/display/learningcenter/Change+Qserver+Memory+Setting

I increased instance3_searchindexer_memory from 900 to 4000. For the time being, I do not seem to have a similar problem but let's see, it is just the second day of this solution.