Ticket #148 (closed bug: fixed)

Opened 6 months ago

Last modified 3 months ago

Manager Unable to Proceed Beyond Cloning Step

Reported by: kindlund Assigned to: kindlund
Priority: high Milestone: 1.1
Component: HoneyClient::Manager Version: 1.02
Severity: none Keywords: manager, clone, process, performance, slow
Cc: mluft@informatik.uni-mannheim.de

Description

Continuation originally, from ticket #135.

Report by mluft:

I watched the dir where the cloned VMs are stored, and the new clone of
 the master vm is created, but the performance is just about 1MiB/Minute.
 That's why i thought it was stucked after 1,5 days ;-)

Infos about the system: * There was no more output after this message 
(not weird with that new informations ;) ) * P4, 2,6Ghz, 1GB RAM, 60GB HD,
 40GB available * I still run Ubuntu 7.10 Server * When i log in with 
vmware console remote, i don't see the cloned vm…

I think there's another problem now about the bad write performance… 

1. To be clear, when you logged onto the VMware Console, did you log in as root? Sometimes, you may have to even start the VMware Console application as root, in order to accomplish this (especially if it's a local login).

2. Also, do you see any type of DHCP log information in your /var/log/messages file? Can you confirm that this file exists?

Attachments

error_honeyClient.txt (380.4 kB) - added by mluft@informatik.uni-mannheim.de on 03/21/08 16:17:36.
clone.vmx (1.4 kB) - added by mluft@informatik.uni-mannheim.de on 03/22/08 08:17:19.
master.vmx (1.4 kB) - added by mluft@informatik.uni-mannheim.de on 03/22/08 08:17:42.
data.zip (3.5 kB) - added by mluft@informatik.uni-mannheim.de on 03/25/08 17:48:51.
error2.txt (108.4 kB) - added by mluft@informatik.uni-mannheim.de on 03/27/08 04:17:37.
unit-tests.txt (8.5 kB) - added by mluft@informatik.uni-mannheim.de on 03/28/08 05:17:18.

Change History

03/07/08 17:13:27 changed by mluft@informatik.uni-mannheim.de

Hi,

ad 1: i enabled the root account and logged in to make sure, and as root i saw 1 running clone and 1 paused - maybe one from an earlier try. the running VM has IP 10.0.0.130 assigned, but no default gw. The output of upcoming honeyclient is:

Starting up Agent - (Hit CTRL-C multiple times to exit.)
/home/Administrator/honeyclient/bin/bootstrap_agent.sh: line 11: /cygdrive/c/Pro
gram Files/VMware/VMware Tools/VMip.exe: No such file or directory
IP =
/home/Administrator/honeyclient/bin/bootstrap_agent.sh: line 15: [: =: unary ope
rator expected

ad 2: The file exists, has permissions 644 and does not contain any dhcp informations

thx one more time…

03/07/08 17:26:57 changed by kindlund

Okay, there are a couple of things you need to verify then.

1. Did you install VMware Tools on your master VM? If so, we assume there's an executable located in:

C:\Program Files\VMware\VMware Tools\VMip.exe

If that's not the default location of the VMip.exe, then you'll probably want to edit your ~/honeyclient/bin/bootstrap_agent.sh script inside the master VM to point to that executable. Remember: When changing anything in the master VM, you'll need to make sure all other clones are powered off and unregistered; also, you'll need to make sure the master VM's disk is set to persistant mode — otherwise, any change you make in the master VM will not get saved properly.

2. The other issue is that the cloned VM should have a default gateway of 10.0.0.254. If that's not getting assigned to this clone, then 1 of 2 things happened: (1) you didn't set the master VM to the Host-Only network or (2) you didn't edit the /etc/vmware/vmnet1/dhcpd/dhcpd.conf file on the host system (see step 2, in this section: http://www.honeyclient.org/trac/wiki/UserGuide#ConfigureVMwareServer)

Let me know if this helps,

— Darien

03/07/08 17:58:16 changed by mluft@informatik.uni-mannheim.de

Hi,

i rely too much on you, i should have seen that the path "program files" does not apply to my german windows installation ;-))

ad 2: Seems to be fixed by 1.

I'll for some results and report here :)

Thx one more time…

03/09/08 18:48:11 changed by kindlund

Okay, glad to hear it. If you run into a related issue, feel free to post in this ticket. Once you get the system operation, please let us know and we'll close this ticket out.

Regards,

— Darien

03/10/08 04:32:46 changed by mluft@informatik.uni-mannheim.de

Hi, i let the whole thing about 1,5 days up and running and the same problem: There's no progress. There's 1 booted clone, but no activity.

No Syslog-entries, no drone errors, no errors in the serverprocess.

Maybe you could give me the exact versions of your setup, e.g. for gentoo, perl, etc. and i will try to set it up just exactly like you?

Cheers, Matthias

(follow-up: ↓ 7 ) 03/10/08 10:43:03 changed by kindlund

Hi Matthias,

It's a little unclear from your comment as to what the problem could be. We'll go through it, step-by-step:

1. While this is all running, can you start up VMware Server Console as root and connect to the actively running clone?

2. While inside the cloned VM, do you see the following line appear inside the Cygwin shell?

URL: http://0.0.0.0:9000/HoneyClient/Agent

3. Can you confirm that the clone VM is currently connected to the Host Only network (vmnet1)?

4. Can you confirm that the firewall VM (v3) is also currently running?

5. On the host system, can you paste the last couple of lines of the StartManager.pl process running?

6. Have you tried providing a URL or set of URLs manually to the StartManager.pl process? (For example, --url_list urls.txt).

7. Lastly, you had indicated that there were no syslog entries. In reality, there should be syslog entries on the host system from both the Agent and the Manager processes. If you're seeing none of this information in your syslog, then you may need to verify that your syslog daemon is configured to accept all remote UDP syslog traffic on 10.0.0.1 port 514 AND is logging ALL messages to the /var/log/messages file — this includes DEBUG level messages. To give you an idea of how to accomplish this, here's an example using sysklogd:

http://www.honeyclient.org/trac/wiki/UserGuide/ConfigureSysklogd

Please let me know more information about these answers and I can try to tailor the troubleshooting further.

— Darien

(in reply to: ↑ 6 ) 03/10/08 14:21:17 changed by mluft@informatik.uni-mannheim.de

Hey ya,

ok, i'm one step further, the system tries to visit some urls. The additional syslog config from http://www.honeyclient.org/trac/wiki/UserGuide/ConfigureSysklogd was missing. I did it in an early setup, and a link to this guide in the actual userguide would be great.

The next problem refers to my german windows installation:

ERROR [HoneyClient::Agent::Driver::Browser::IE::drive] (lib/HoneyClient/Agent/Driver/Browser/IE.pm:440) - Error: Unable to execute C:\Program Files\Internet Explorer\iexplore.exe

This is caused by my german windows installation, and i would fix this in the code, but i think this is a neverending story ;-)

Is there a central config where i'm abled to prove paths etc.?

Thx&Cheers, Matthias

03/10/08 15:13:04 changed by kindlund

  • status changed from new to assigned.

Hi Matthias,

Yes, that explains it. Okay, there is a central configuration file, that you can edit inside the master VM. Specifically, the file is:

~/honeyclient/etc/honeyclient.xml

In there, you'll find the following entry:

<HoneyClient>
    <Agent>
        <Browser>
            <IE>
                <process_exec>
                    C:\Program Files\Internet Explorer\iexplore.exe
                </process_exec>
            </IE>
        </Browser>
    </Agent>
</HoneyClient>

You might want to look throughout that .xml file and check to see if there are any other "Program Files" references that you might need to change. Also, because the data is stored in XML format, you may need to use the XML notation to represent Unicode characters properly. If you have trouble with this, let me know and I can try to provide the corresponding mappings.

— Darien

03/11/08 04:41:35 changed by mluft@informatik.uni-mannheim.de

Hi,

this worked great, but - not really suprising - there's a new errormessage ;-)

2008-03-11 09:19:13  INFO [HoneyClient::Manager::runSession] (lib/HoneyClient/Manager.pm:767) - Saving fingerprint to 'fingerprint.dump'.
2008-03-11 09:19:13  INFO [HoneyClient::Manager::runSession] (lib/HoneyClient/Manager.pm:779) - Archiving VM...
2008-03-11 09:19:33  INFO [HoneyClient::Manager::VM::snapshotVM] (lib/HoneyClient/Manager/VM.pm:4418) - Snapshotting VM (/home/honeyusr/vm/clone/0f9da33d3c8cd92a0e11c45844/master.vmx) to (/home/honeyusr/vm/snapshots/0f9da33d3c8cd92a0e11c45844-20080311T081933.tar.gz).
Starting new session...
2008-03-11 09:19:36  INFO [HoneyClient::Manager::VM::Clone::new] (lib/HoneyClient/Manager/VM/Clone.pm:885) - Setting VM (/home/honeyusr/vm/master/master.vmx) as master.
2008-03-11 09:19:52  INFO [HoneyClient::Manager::VM::Clone::_init] (lib/HoneyClient/Manager/VM/Clone.pm:580) - Quick cloning master VM (/home/honeyusr/vm/master/master.vmx).
/bin/tar: 0f9da33d3c8cd92a0e11c45844/master.vmem: file changed as we read it
2008-03-11 09:20:08  WARN [HoneyClient::Manager::VM::__ANON__] (lib/HoneyClient/Manager/VM.pm:4481) - Could not snapshot VM to (/home/honeyusr/vm/snapshots/0f9da33d3c8cd92a0e11c45844-20080311T081933.tar.gz). (256: )
2008-03-11 09:20:12 ERROR [HoneyClient::Util::SOAP::_handleFault] (lib/HoneyClient/Util/SOAP.pm:283) - Error occurred during processing. HoneyClient::Manager::VM->snapshotVM(): Could not snapshot VM to (/home/honeyusr/vm/snapshots/0f9da33d3c8cd92a0e11c45844-20080311T081933.tar.gz).
HoneyClient::Manager::VM->snapshotVM(): {'err' => bless( {'errNo' => '256','errStr' => ''}, 'err' )}
HoneyClient::Util::SOAP->handleFault(): Error occurred during processing.
HoneyClient::Manager::VM->snapshotVM(): Could not snapshot VM to (/home/honeyusr/vm/snapshots/0f9da33d3c8cd92a0e11c45844-20080311T081933.tar.gz).
HoneyClient::Manager::VM->snapshotVM(): {'err' => bless( {'errNo' => '256','errStr' => ''}, 'err' )}
Killed

The only problem i could imagine is that there are too restrictive permissions, but i checked it, and there's no problem. For testing purposes i set the permissions to 777, same problem. Suprisingly, there is a file named 0f9da33d3c8cd92a0e11c45844-20080311T081933.tar.gz in the mentioned directory….

03/11/08 10:57:15 changed by kindlund

Hi Matthias,

Okay, so the problem here, I believe, is that when you executed StartManager.pl, the master VM was not properly finalized before you started. Specifically, I'm thinking that your config.version, virtualHW.version, and/or .mode wasn't set properly in your /home/honeyusr/vm/master/master.vmx file.

Specifically, make sure you perform the following operations on the master VM, as described in this section: http://www.honeyclient.org/trac/wiki/UserGuide#FinalizeMasterVMSettings

In parallel, I could help and take a look at your master.vmx file, if you could attach it to this ticket in your next post. If I spot any problems, I could clean them up and send you the fixed version of this file.

Regards,

— Darien

03/14/08 04:36:31 changed by mluft@informatik.uni-mannheim.de

Hi,

i checked the mastervm including the config and could not find any differences to the guide. I attached the master.vmx, would be great if you would have a look at it.

Cheers, Matthias

#!/usr/bin/vmware
config.version = "7"
virtualHW.version = "3"
scsi0.present = "TRUE"
memsize = "384"
ide0:0.present = "TRUE"
ide0:0.fileName = "/home/honeyusr/vm/master/master.vmdk"
ide0:0.writeThrough = "TRUE"
ide1:0.present = "TRUE"
ide1:0.fileName = "E:"
ide1:0.deviceType = "cdrom-raw"

ide0:0.mode ="persistent"

floppy0.startConnected = "FALSE"
floppy0.fileName = "/dev/fd0"
Ethernet0.present = "TRUE"
displayName = "master"
guestOS = "winxppro"
priority.grabbed = "normal"
priority.ungrabbed = "normal"
powerType.powerOff = "hard"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "hard"

ide0:0.redo = ""
ethernet0.addressType = "generated"
uuid.location = "56 4d 09 4a bc 70 fa 16-41 21 77 2d e8 2d c2 53"
uuid.bios = "56 4d 09 4a bc 70 fa 16-41 21 77 2d e8 2d c2 53"
checkpoint.vmState = ""
ethernet0.generatedAddress = "00:0c:29:2d:c2:53"
ethernet0.generatedAddressOffset = "0"

uuid.action = "create"

ide1:0.startConnected = "FALSE"
tools.syncTime = "TRUE"

Ethernet0.connectionType = "hostonly"

floppy0.present = "FALSE"

ide0:0.deviceType = "ata-hardDisk"

03/14/08 11:52:51 changed by kindlund

Hi Matthias,

Okay, there were couple of issues that I'm not sure would affect this situation, but it's worth a shot testing them out. Here's my revisions to your .vmx file:

#!/usr/bin/vmware
config.version = "7"
virtualHW.version = "3"

#OS
displayName = "master"
guestOS = "winxppro"

#Virtual Disk
ide0.present = "TRUE"
ide0:0.present = "TRUE"
ide0:0.mode ="persistent"
ide0:0.fileName = "master.vmdk"

#Memory size
memsize = "384"

#Nvram
nvram = "master.nvram"

#CD-ROM
ide1:0.present = "TRUE"
ide1:0.fileName = "/dev/cdrom"
ide1:0.deviceType = "cdrom-raw"
ide1:0.startConnected = "FALSE"

#Floppy
floppy0.present = "FALSE"
floppy0.fileName = "/dev/fd0"
floppy0.fileType = "device"
floppy0.startConnected = "FALSE"

#Networking
Ethernet0.present = "TRUE"
Ethernet0.connectionType = "hostonly"

#Misc
RemoteDisplay.depth = 16
log.fileName = "master.log"

localConsole = "FALSE"
draw = "gdi"
RemoteDisplay.bpp = 8
uuid.location = "56 4d 09 4a bc 70 fa 16-41 21 77 2d e8 2d c2 53"
tools.syncTime = "TRUE"

usb.generic.autoconnect = "FALSE"
uuid.bios = "56 4d 09 4a bc 70 fa 16-41 21 77 2d e8 2d c2 53"

priority.grabbed = "normal"
priority.ungrabbed = "normal"

usb.present = "FALSE"

Ethernet0.addressType = "generated"
uuid.action = "create"
ethernet0.generatedAddress = "00:0c:29:2d:c2:53"
ethernet0.generatedAddressOffset = "0"

sound.present = "FALSE"

undopoints.seqNum = "0"
ide0:0.redo = ""
undopoint.restoreFromCheckpoint = "FALSE"
undopoint.checkpointedOnline = "FALSE"

undopoint.protected = "FALSE"
gui.restricted = "FALSE"

ide0:0.deviceType = "ata-hardDisk"

numvcpus = "1"

Okay, so let me know if this file works for you. If it does, bonus points if you can figure out the directives that caused the problems in the first place. ;)

If it doesn't work, let me know and we can try another solution.

Regards,

— Darien

03/19/08 12:39:10 changed by mluft@informatik.uni-mannheim.de

Hi,

sorry for my late reply.

Still the same problem with the new master.vmx. Above the errormessage very long numbers of digits are printed, declared as "REG_BINARY", does this output matter?

Thx one more time..

Matthias

03/19/08 12:56:10 changed by kindlund

Hrm, it would be helpful if you could provide me with the exact error message; if it's too long to paste into ticket box, save it as a file and attach it to the ticket.

Thanks,

— Darien

03/21/08 16:17:36 changed by mluft@informatik.uni-mannheim.de

  • attachment error_honeyClient.txt added.

03/21/08 16:19:00 changed by mluft@informatik.uni-mannheim.de

Hi,

I attached the full error message.

Cheers, Matthias

03/21/08 16:56:54 changed by kindlund

Great! Okay, I'm looking through it now. One thing does appear rather quickly:

2008-03-21 20:56:15 ERROR [HoneyClient::Manager::Database::_AUTOLOAD] (lib/HoneyClient/Manager/Database.pm:233) - Error: RPC communications failure.
2008-03-21 20:56:15  WARN [HoneyClient::Manager::runSession] (lib/HoneyClient/Manager.pm:670) - Failure Inserting Client Object:
Error: RPC communications failure. at lib/HoneyClient/Manager.pm line 962

Are you running the Drone web service? Or are you trying to run honeyclients without it?

More soon,

— Darien

03/21/08 17:05:44 changed by kindlund

Also, after running this, could you please attach the following 2 files to this ticket?

/home/honeyusr/vm/master/master.vmx /home/honeyusr/vm/clone/07888a49c481c2757e48f6a01b/master.vmx (Call this file "clone.vmx" when you upload it)

Thanks,

— Darien

03/22/08 08:16:56 changed by mluft@informatik.uni-mannheim.de

Hi,

drone is definitely running and i attached the two files… I see a problem yet, in clone.vmx the hw-version is set to 4, if this does matter.

Cheers, Matthias

03/22/08 08:17:19 changed by mluft@informatik.uni-mannheim.de

  • attachment clone.vmx added.

03/22/08 08:17:42 changed by mluft@informatik.uni-mannheim.de

  • attachment master.vmx added.

03/22/08 15:24:33 changed by kindlund

  • type changed from issue to bug.

Hi Matthias,

I think you may have stumbled upon a legitimate bug, actually. If you look at the master configuration file that you specified:

http://www.honeyclient.org/trac/attachment/ticket/148/master.vmx

Line 12 reads:

ide0:0.mode ="persistent"

Can you change that line to read:

ide0:0.mode = "persistent"

(Place 1 space between the equals and first double-quote characters.)

Then, save it and re-run StartManager.pl and let me know if the new clone has any more problems.

Regards,

— Darien

03/25/08 12:05:50 changed by kindlund

Hi Matthias,

Did that last suggestion help? Please let us know.

Thanks,

— Darien

03/25/08 12:11:00 changed by mluft@informatik.uni-mannheim.de

Hi,

I'm currently separated from my honeyboxes, but as soon as i know i'll report!

Thanks, Matthias

03/25/08 17:11:29 changed by mluft@informatik.uni-mannheim.de

Hi,

still the same problem… any more information? ;-)

Although the client isn't running yet, i want to thank you really much for you great support, i never experienced such a great and fast help :-) Thank you very much!

Cheers, Matthias

03/25/08 17:20:46 changed by kindlund

Okay, if you properly changed that line, then StartManager.pl should have updated the "ide0:0.mode" line within your clone. Open up your most recently created .vmx file inside the cloned VM directory and tell me what your "ide0:0.mode" line says.

For example: (/home/honeyusr/vm/clone/07888a49c481c2757e48f6a01b/master.vmx), but use the newer clone directory.

Does it say?

ide0:0.mode = "persistent"

Or does it say?

ide0:0.mode = "undoable"

If you can, please paste the EXACT characters listed on that line — feel free to post the entire .vmx file, if you'd prefer.

Thanks,

— Darien

03/25/08 17:27:30 changed by mluft@informatik.uni-mannheim.de

Hi,

it's undoable:

#Virtual Disk
ide0.present = "TRUE"
ide0:0.present = "TRUE"
ide0:0.mode = "undoable"
ide0:0.fileName = "/home/honeyusr/vm/master/master.vmdk"

if it helps i will post again the entire file…

thanks, Matthias

03/25/08 17:43:06 changed by kindlund

Yeah, can you post the entire file?

Thanks,

— Darien

03/25/08 17:48:51 changed by mluft@informatik.uni-mannheim.de

  • attachment data.zip added.

03/25/08 17:50:21 changed by mluft@informatik.uni-mannheim.de

Hi,

i attached data.zip, which contains:

*clone1.vmx: first clone, which is cloned successfully *clone2.vmx: the one which causes the error *master.vmx: the master.vmx *error.txt: last part of the error messages,

Thanks, Matthias

03/25/08 20:18:05 changed by kindlund

After looking closer at the clone1.vmx and clone2.vmx files, it looks like you may have reversed your descriptions — are you sure that clone1.vmx was the first one that cloned successfully?

Anyway, one fundamental issue that I've noticed in your clone2.vmx file is this:

virtualHW.version = "4"

That should be:

virtualHW.version = "3"

When you create the clone, do you perform any other actions on the clone, such as installing VMware Tools or upgrade the virtual hardware? When the clone is running, can you check the .vmx file and see if the virtualHW.version setting has changed?

Also, what version of VMware Server are you using? v1.0.5?

Lastly, are your /home/honeyusr/vm/* directories all located on the same disk, or are you spanning multiple disks/NFS shares?

Lastly, as a long shot, try renaming master.vmx to master.cfg and re-run StartManager.pl to see if this helps.

Regards,

— Darien

03/25/08 20:22:54 changed by kindlund

Matthias,

One other question, can you also paste the contents of your /home/honeyusr/vm/master/master.vmdk file?

Specifically, I'm curious if it has an entry like this:

ddb.virtualHWVersion = "3"

Or like this:

ddb.virtualHWVersion = "4"

Thanks,

— Darien

(follow-up: ↓ 30 ) 03/26/08 03:52:01 changed by mluft@informatik.uni-mannheim.de

Hi,

When you create the clone, do you perform any other actions on the clone, such as installing VMware Tools or upgrade the virtual hardware? When the clone is running, can you check the .vmx file and see if the virtualHW.version setting has changed?

What do you mean with "When you create the clone"? The Clone is created just in the process of the running StartManager.pl?

I'm using VMware Server 1.0.4 for some reason - i think the last time i set it up 1.0.4 were referenced in your user guide.

Everything is located on one single disk & partition.

Should i completely rename master.vmx to master.cfg or just copy it?

Here's the master.vmdk whith HWVersion = "4":

# Disk DescriptorFile
version=1
CID=f1825903
parentCID=ffffffff
createType="twoGbMaxExtentSparse"

# Extent description
RW 4192256 SPARSE "master-s001.vmdk"
RW 4192256 SPARSE "master-s002.vmdk"
RW 4192256 SPARSE "master-s003.vmdk"
RW 6144 SPARSE "master-s004.vmdk"

# The Disk Data Base
#DDB

ddb.toolsVersion = "6532"
ddb.adapterType = "ide"
ddb.geometry.sectors = "63"
ddb.geometry.heads = "16"
ddb.geometry.cylinders = "12483"
ddb.virtualHWVersion = "4"

(in reply to: ↑ 29 ) 03/26/08 10:09:59 changed by kindlund

Replying to mluft@informatik.uni-mannheim.de:

Here's the master.vmdk whith HWVersion = "4": {{{ ddb.virtualHWVersion = "4" }}}

Okay, I think I know what the issue is now. Even though we're specifying virtualHW.version = "3" in the master.vmx file, that ddb.virtualHWVersion directive is overriding it.

Let's try this: Make sure you have a backup of your current /home/honeyusr/vm/master VM, then edit your /home/honeyusr/vm/master/master.vmdk file and change this line:

ddb.virtualHWVersion = "4"

to read:

ddb.virtualHWVersion = "3"

Then run StartManager.pl and let me know if that still causes the error:

/bin/tar: 07888a49c481c2757e48f6a01b/master.vmem: file changed as we read it

Thanks,

— Darien

03/27/08 04:17:07 changed by mluft@informatik.uni-mannheim.de

Hi,

the error still exists with ddb.virtualHWVersion = "3".

I attached the errormessages as error2.txt, but i think it's the same output.

Thanks, Matthias

03/27/08 04:17:37 changed by mluft@informatik.uni-mannheim.de

  • attachment error2.txt added.

03/27/08 08:47:40 changed by kindlund

Yeah, it looks the same, with the same tar error message.

Now that the StartManager.pl process has been killed, can you check to see that your dd.virtualHWVersion is still set to 3? I'm curious if VMware Server changed this value during its operations.

One of the last things we can try, is copying the master.vmx file to master.cfg and then updating your master_vm_config reference in ~/honeyclient/etc/honeyclient.xml to reflect .cfg instead of .vmx. Then, see if running StartManager.pl causes that error message to disappear. (You'll want to make sure virtualHW.version in the master.cfg file is set to 3 and dd.virtualHWVersion in the master.vmdk file is also set to 3.)

Regards,

— Darien

03/27/08 08:59:07 changed by kindlund

Hi Matthias,

If the last suggestion I mentioned did not work and you still see that error message, then here's one last test you can try. We developed a suite of unit tests that you should be able to run and then provide us with the output — these tests should exhaustively identify any problems with the VM replication code.

On the host system, type the following and please answer YES to both questions.

$ cd ~/honeyclient
$ perl -Ilib t/honeyclient_manager_vm.t
...
# About to run basic unit tests.
# Note: These tests *expect* VMware Server or VMware GSX to be installed and running on this system beforehand.
# Do you want to run basic tests? [yes] yes
...
# About to run extended tests.
# Note: These extended tests will take *significant* time to complete (10-30 minutes).
# Do you want to run extended tests? [no] yes
...

Thanks,

— Darien

03/28/08 05:16:42 changed by mluft@informatik.uni-mannheim.de

Hi,

both versions are stillcset to 3 after the StartManager-process dies.

Luckily, the unittests report some errors. Unluckily, i don't think they are very important ;-) I attached unit-tests.txt.

Could you give me a list of your environment, e.g. your used distro, vmware server version and so on, so i could test if i get it working this way?

Thanks, Matthias

03/28/08 05:17:18 changed by mluft@informatik.uni-mannheim.de

  • attachment unit-tests.txt added.

03/28/08 10:40:40 changed by kindlund

Hi Matthias,

I think the unit tests reveal a fundamental problem with the host system that you're running this on.

1) Do you have any other VMs running on this system, aside from the Firewall VM and a single Cloned VM?

2) Do you have any other load-intensive processes on the host system? For example, what is the load average of the system?

In order to run honeyclients with full capabilities, we typically run the code on idle systems that have 2.4 Ghz CPUs with at least 2 GB of RAM. In this case, I'm thinking that either your CPU is throttled or the combined processes eat up too much RAM, causing some of the processes' RAM to get paged to your swap file (you may be able to see this activity by running top while running StartManager.pl).

In any event, I can offer a workaround — it involves having the manager code skip the archive process, which happens when a cloned VM is discovered to be compromised. Essentially, the manager tries to suspend the running clone and archive a copy of it to /vm/snapshots while also trying to create a new clone. I'm thinking that this degree of activity on your system is just too much for it to handle at any one time.

So, here's the workaround. On the host system, type:

$ cd ~/honeyclient/
$ vi lib/HoneyClient/Manager.pm

Around line 782 in this file, you should see this code:

                    # Archive the VM.
                    $LOG->info("Archiving VM...");
                    $vm->archive();

Comment these lines out, so that it reads:

                    # Archive the VM.
                    #$LOG->info("Archiving VM...");
                    #$vm->archive();

Then, try re-running StartManager.pl and let me know if this resolves the issue.

Regards,

— Darien

03/31/08 15:56:36 changed by kindlund

Hi Matthias,

Any luck with the patch I mentioned earlier? Please let us know.

Thanks,

— Darien

03/31/08 16:25:35 changed by mluft@informatik.uni-mannheim.de

Hi,

sorry for late reply, as always there was lot of work ;-)

Commenting out the archiving seems to work! But there's still the following error:

2008-03-31 22:15:55 ERROR [HoneyClient::Manager::Database::_AUTOLOAD] (lib/HoneyClient/Manager/Database.pm:233) - Error: RPC communications failure.
2008-03-31 22:15:56  WARN [HoneyClient::Manager::runSession] (lib/HoneyClient/Manager.pm:670) - Failure Inserting Client Object:
Error: RPC communications failure. at lib/HoneyClient/Manager.pm line 962

Maybe logging to the database is not possible? Our studied has to be finished soon, and i would love to write something about HoneyClient so i need some reports ;-)

Cheers, Matthias

(follow-up: ↓ 43 ) 03/31/08 16:57:46 changed by kindlund

Hi Matthias,

Great! Glad to hear it worked.

Now, about the new error that you're getting. It looks like you don't have the Drone webservice up and running, is this correct?

If so, you'll want to update your ~/etc/honeyclient.xml file on the host system and make sure the following is set:

<HoneyClient>
    <Manager>
        <Database>
            <enable description="Enables database operations. 1 enables, 0 disables." default="1">
                0
            </enable>
        </Database>
    </Manager>
</HoneyClient>

Let me know if this helps.

So far, I see a list of improvements we can make, based upon this ticket:

  • Specify RAM requirements in the UserGuide
  • Provide a configurable option to enable/disable VM archiving
  • Improve the current setMasterVM() logic that's responsible for changing low-level settings in the master VM's configuration file (.vmx/.cfg)

We'll get to work on that as well.

Regards,

— Darien

03/31/08 17:03:41 changed by mluft@informatik.uni-mannheim.de

Hi,

indeed as i mentioned above the drone service is up and running. But if a get some reports without a database too i'm really happy with a simple textoutput?

Thanks&Cheers, Matthias

03/31/08 17:10:07 changed by kindlund

Okay, try without the drone and make sure it's all working properly. Yes, by default, StartManager.pl will write two files out: "manager.dump" and "compromise.dump", so you should be able to get text output in this fashion.

If you want the database communication to work properly, can you verify that you're able to browse to "http://<host_system_ip>:3000" and obtain the Drone webservice login page?

If not, make sure the ruby WEBrick process is running properly by calling:

$ ps ax | grep ruby

Regards,

— Darien

04/22/08 02:30:28 changed by kindlund

Hi Matthias,

Just wanted to follow-up with you, regarding this issue. Has it been resolved? Please let us know.

Thanks,

— Darien

04/22/08 02:39:45 changed by mluft@informatik.uni-mannheim.de

Hi Darien,

thanks for your interest, everything was working fine and i was able to complete the necessary tests.

As i mentioned: GREAT Support, i want to thank you one more time!

Matthias

(in reply to: ↑ 38 ) 05/27/08 17:24:09 changed by kindlund

  • status changed from assigned to closed.
  • resolution set to fixed.

Replying to kindlund:

So far, I see a list of improvements we can make, based upon this ticket: * Specify RAM requirements in the UserGuide

Fixed UserGuide.

* Provide a configurable option to enable/disable VM archiving

Added an archive_upon_suspend option to the etc/honeyclient.xml configuration file.

* Improve the current setMasterVM() logic that's responsible for changing low-level settings in the master VM's configuration file (.vmx/.cfg)

Fixed in r1590.

Closing ticket; all fixes have been committed to trunk.


Add/Change #148 (Manager Unable to Proceed Beyond Cloning Step)




Change Properties
Action