[Xymon] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client

[SebA]

I have noticed a memory and thread leak in the Windows Virtual Disk service
- or vds.exe from the process list. This service is set to Manual by
default and gets started when starting the XymonPSClient service on Windows
2008 R2. It starts off using 32 threads, but grows forever, the virtual
memory and physical memory (which starts under 6 MB) used slowly grow too.

Has anyone noticed this problem and do they have a solution to it?

On Windows Server 2012, the Virtual Disk service does not get started by
Xymon Windows Powershell client, so it's obviously not started directly by
the client, but by a request for information. I looked into known memory
leaks for vds.exe and the one I found was not applicable to 2008 R2, only
2008 and Vista. The version of vds.exe we have is later than the version
with the fix: 6.1.7601.17514 from 2010.

The quick workaround is simply to stop the Virtual Disk Service - but as
soon as the PSclient has used it, it does not stop properly and terminates
unexpectedly, so one needs to set it not to restart on crash, or stop it
immediately after starting, or to Disabled. But it would be good to know
why it is starting, not stopping, and leaking threads / memory!

Kind regards,

SebA

[Beck, Zak]

Hi Seb

I’ve got hundreds of VMs running the Powershell client on Windows Server 2008R2, 2012R2 and 2016 and I don’t see this issue. I just checked a couple of servers, vds.exe is between 2MB and 20MB and around 21 threads. Same version of vds.exe too.

I’m wondering if function XymonDiskPart has something to do with it. This only runs on slow scans (every 6 hours-ish by default). This runs ‘diskpart’ and then

“list disk”
For each disk returned, “select disk {n}” and then “detail disk”

Maybe you could try this from the command prompt on a server with vds stopped and see if it restarts it. You could also try commenting out this line (3875, towards the end) in the client and see what happens (insert a # at the beginning of the line):

$script:diskpartData = XymonDiskPart


Alternatively, it could be the disk utilisation code which is using Windows API calls in kernel32.dll – FindFirstVolume, FindNextVolume, FindVolumeClose, GetVolumeInformation, GetVolumePathNamesForVolumeNameW, GetDiskFreeSpaceEx and GetDriveType. The MSDN documentation for these doesn’t mention the Virtual Disk Service but I guess they could be using it.

Zak

From: SebA [mailto:***@syntec.co.uk]
Sent: Tuesday, 13 March 2018 18:14
To: Beck, Zak <***@accenture.com>; Xymon Mailing List <***@xymon.com>
Subject: [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client

I have noticed a memory and thread leak in the Windows Virtual Disk service - or vds.exe from the process list. This service is set to Manual by default and gets started when starting the XymonPSClient service on Windows 2008 R2. It starts off using 32 threads, but grows forever, the virtual memory and physical memory (which starts under 6 MB) used slowly grow too.

Has anyone noticed this problem and do they have a solution to it?

On Windows Server 2012, the Virtual Disk service does not get started by Xymon Windows Powershell client, so it's obviously not started directly by the client, but by a request for information. I looked into known memory leaks for vds.exe and the one I found was not applicable to 2008 R2, only 2008 and Vista. The version of vds.exe we have is later than the version with the fix: 6.1.7601.17514 from 2010.

The quick workaround is simply to stop the Virtual Disk Service - but as soon as the PSclient has used it, it does not stop properly and terminates unexpectedly, so one needs to set it not to restart on crash, or stop it immediately after starting, or to Disabled. But it would be good to know why it is starting, not stopping, and leaking threads / memory!

Kind regards,

SebA

________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com

[Alessandro Tinivelli]

Good morning,

i just noticed a memory leak caused bur our beloved ps client on win 2008r2: task manager reported a process “powershell” launched by System using 1.7GB.
I uninstalled the ps client and the memory was immediately released.

Version in use was 2.19 (not so young).

Alessandro

[Beck, Zak]

Hi Alessandro

I think this is going to incredibly hard to track down as I haven’t had any other reports like this. I personally have the client running on 3-400 servers and am not aware of issues like this.

You sent me a couple of logs (thanks), could you explain the configuration differences between the server having the issue and others (client-local)?

Thanks

Zak

From: Xymon [mailto:xymon-***@xymon.com] On Behalf Of Alessandro Tinivelli
Sent: Friday, 23 March 2018 11:08
To: Xymon Mailing List <***@xymon.com>
Subject: [Xymon] [EXCH Prio5]R: [EXCH Prio5]Re: [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client

Good morning,

i just noticed a memory leak caused bur our beloved ps client on win 2008r2: task manager reported a process “powershell” launched by System using 1.7GB.
I uninstalled the ps client and the memory was immediately released.

Version in use was 2.19 (not so young).

Alessandro

________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com

[SebA]

Hi Zak,

Thanks for your swift reply and sorry for my own delay! It's good to know
you have such extensive use of the PS client without this issue.

I think you are probably right. These issue has only appeared on servers
using Windows Cluster Failover, where various iSCSI disks are clustered.

For the record, my workaround of disabling the Virtual Disk Service caused
no bad effects on the server, but it does prevent diskpart from running. I
set the service back to Manual, and it started as soon as I ran diskpart
from an elevated Command Prompt. Exiting diskpart causes the VDS service
and proces to stop.

When the Virtual Disk Service was set to manual, and started just by me
running diskpart.exe, the client output has:

[diskpart]
diskpart:::Clustered Unknown

When the service is disabled, the output is just an empty section:

[diskpart]

When VDS is started by me restarting the XymonPSclient service and it being
on manual, the output is e.g.:

[diskpart]
diskpart:Disk 0:278 GB:Not Clustered
diskpart:Disk 1:1024 MB:Clustered Active
diskpart:Disk 2:25 GB:Clustered Active
diskpart:Disk 3:500 GB:Clustered Active
diskpart:Disk 4:300 GB:Clustered Inactive
diskpart:Disk 5:1024 MB:Clustered Inactive
diskpart:Disk 6:500 GB:Clustered Inactive
diskpart:Disk 7:200 GB:Clustered Inactive
diskpart:Disk 8:1024 MB:Clustered Inactive
diskpart:Disk 9:1024 MB:Clustered Inactive
diskpart:Disk 10:200 GB:Clustered Inactive
diskpart:Disk 11:200 GB:Clustered Inactive
diskpart:Disk 12:500 GB:Clustered Active
diskpart:Disk 13:5120 MB:Clustered Active
diskpart:Disk 14:500 GB:Clustered Inactive

Compared with VDS running because of me running diskpart and running:

“list disk”

For each disk returned, “select disk {n}” and then “detail disk”

for each disk (which does not leak memory) vds.exe uses about an extra MB
of RAM when running due to the PS client.

I have also now commented out the line you suggest (' $script:diskpartData
= XymonDiskPart ') and now when I restart the XymonPSclient service with
VDS set to Manual it does not get restarted and left running (or restarted
at all). So this looks like a better workaround than just disabling the
service, but I will need to continue to monitor that.

Is the XymonDiskPart information used for anything? I mean, can it be used
for anything without writing plug-ins?

We do not have memory leaks in powershell.exe on these servers (although I
think I may have seen it on another server), so the issue above is not
related to Alessandro's issue.

Thanks again for your help Zak, and I hope this knowledge is of benefit to
you or others too.


Kind regards,

SebA

Post by Beck, Zak
Hi Seb
I’ve got hundreds of VMs running the Powershell client on Windows Server
2008R2, 2012R2 and 2016 and I don’t see this issue. I just checked a couple
of servers, vds.exe is between 2MB and 20MB and around 21 threads. Same
version of vds.exe too.
I’m wondering if function XymonDiskPart has something to do with it. This
only runs on slow scans (every 6 hours-ish by default). This runs
‘diskpart’ and then
“list disk”
For each disk returned, “select disk {n}” and then “detail disk”
Maybe you could try this from the command prompt on a server with vds
stopped and see if it restarts it. You could also try commenting out this
line (3875, towards the end) in the client and see what happens (insert a #
$script:diskpartData = XymonDiskPart
Alternatively, it could be the disk utilisation code which is using
Windows API calls in kernel32.dll – FindFirstVolume, FindNextVolume,
FindVolumeClose, GetVolumeInformation, GetVolumePathNamesForVolumeNameW,
GetDiskFreeSpaceEx and GetDriveType. The MSDN documentation for these
doesn’t mention the Virtual Disk Service but I guess they could be using it.
Zak
*Sent:* Tuesday, 13 March 2018 18:14
*Subject:* [External] Memory leak in Windows Virtual Disk service caused
by Xymon Windows Powershell client
I have noticed a memory and thread leak in the Windows Virtual Disk
service - or vds.exe from the process list. This service is set to Manual
by default and gets started when starting the XymonPSClient service on
Windows 2008 R2. It starts off using 32 threads, but grows forever, the
virtual memory and physical memory (which starts under 6 MB) used slowly
grow too.
Has anyone noticed this problem and do they have a solution to it?
On Windows Server 2012, the Virtual Disk service does not get started by
Xymon Windows Powershell client, so it's obviously not started directly by
the client, but by a request for information. I looked into known memory
leaks for vds.exe and the one I found was not applicable to 2008 R2, only
2008 and Vista. The version of vds.exe we have is later than the version
with the fix: 6.1.7601.17514 from 2010.
The quick workaround is simply to stop the Virtual Disk Service - but as
soon as the PSclient has used it, it does not stop properly and terminates
unexpectedly, so one needs to set it not to restart on crash, or stop it
immediately after starting, or to Disabled. But it would be good to know
why it is starting, not stopping, and leaking threads / memory!
Kind regards,
SebA
------------------------------
This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise confidential information. If you have
received it in error, please notify the sender immediately and delete the
original. Any other use of the e-mail by you is prohibited. Where allowed
by local law, electronic communications with Accenture and its affiliates,
including e-mail and instant messaging (including content), may be scanned
by our systems for the purposes of information security and assessment of
internal compliance with Accenture policy.
____________________________________________________________
__________________________
www.accenture.com

[Beck, Zak]

Hi Seb

Thanks for confirming, that’s good news – I was fearing it may be the core disk utilisation functions which would be a much larger headache!

Is the XymonDiskPart information used for anything? I mean, can it be used for anything without writing plug-ins?

The [diskpart] section is used in one of our datacentres where they use clustering and the information is used primarily for reporting. So no, it is not used for anything by default.

I think the best solution here is for me to add a setting in xymonclient_config.xml allowing you to turn off the diskpart function for specific servers. You can then remove your comment and continue to use future updates.

Zak

From: SebA [mailto:***@syntec.co.uk]
Sent: Friday, 30 March 2018 14:04
To: Beck, Zak <***@accenture.com>
Cc: Xymon Mailing List <***@xymon.com>
Subject: Re: [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client

Hi Zak,
Thanks for your swift reply and sorry for my own delay! It's good to know you have such extensive use of the PS client without this issue.
I think you are probably right. These issue has only appeared on servers using Windows Cluster Failover, where various iSCSI disks are clustered.
For the record, my workaround of disabling the Virtual Disk Service caused no bad effects on the server, but it does prevent diskpart from running. I set the service back to Manual, and it started as soon as I ran diskpart from an elevated Command Prompt. Exiting diskpart causes the VDS service and proces to stop.
When the Virtual Disk Service was set to manual, and started just by me running diskpart.exe, the client output has:

[diskpart]

diskpart:::Clustered Unknown
When the service is disabled, the output is just an empty section:

[diskpart]
When VDS is started by me restarting the XymonPSclient service and it being on manual, the output is e.g.:

[diskpart]

diskpart:Disk 0:278 GB:Not Clustered

diskpart:Disk 1:1024 MB:Clustered Active

diskpart:Disk 2:25 GB:Clustered Active

diskpart:Disk 3:500 GB:Clustered Active

diskpart:Disk 4:300 GB:Clustered Inactive

diskpart:Disk 5:1024 MB:Clustered Inactive

diskpart:Disk 6:500 GB:Clustered Inactive

diskpart:Disk 7:200 GB:Clustered Inactive

diskpart:Disk 8:1024 MB:Clustered Inactive

diskpart:Disk 9:1024 MB:Clustered Inactive

diskpart:Disk 10:200 GB:Clustered Inactive

diskpart:Disk 11:200 GB:Clustered Inactive

diskpart:Disk 12:500 GB:Clustered Active

diskpart:Disk 13:5120 MB:Clustered Active

diskpart:Disk 14:500 GB:Clustered Inactive
Compared with VDS running because of me running diskpart and running:
“list disk”
For each disk returned, “select disk {n}” and then “detail disk”

for each disk (which does not leak memory) vds.exe uses about an extra MB of RAM when running due to the PS client.
I have also now commented out the line you suggest (' $script:diskpartData = XymonDiskPart ') and now when I restart the XymonPSclient service with VDS set to Manual it does not get restarted and left running (or restarted at all). So this looks like a better workaround than just disabling the service, but I will need to continue to monitor that.
Is the XymonDiskPart information used for anything? I mean, can it be used for anything without writing plug-ins?

We do not have memory leaks in powershell.exe on these servers (although I think I may have seen it on another server), so the issue above is not related to Alessandro's issue.
Thanks again for your help Zak, and I hope this knowledge is of benefit to you or others too.


Kind regards,

SebA

On 14 March 2018 at 09:21, Beck, Zak <***@accenture.com<mailto:***@accenture.com>> wrote:
Hi Seb

I’ve got hundreds of VMs running the Powershell client on Windows Server 2008R2, 2012R2 and 2016 and I don’t see this issue. I just checked a couple of servers, vds.exe is between 2MB and 20MB and around 21 threads. Same version of vds.exe too.

I’m wondering if function XymonDiskPart has something to do with it. This only runs on slow scans (every 6 hours-ish by default). This runs ‘diskpart’ and then

“list disk”
For each disk returned, “select disk {n}” and then “detail disk”

Maybe you could try this from the command prompt on a server with vds stopped and see if it restarts it. You could also try commenting out this line (3875, towards the end) in the client and see what happens (insert a # at the beginning of the line):

$script:diskpartData = XymonDiskPart


Alternatively, it could be the disk utilisation code which is using Windows API calls in kernel32.dll – FindFirstVolume, FindNextVolume, FindVolumeClose, GetVolumeInformation, GetVolumePathNamesForVolumeNameW, GetDiskFreeSpaceEx and GetDriveType. The MSDN documentation for these doesn’t mention the Virtual Disk Service but I guess they could be using it.

Zak

From: SebA [mailto:***@syntec.co.uk<mailto:***@syntec.co.uk>]
Sent: Tuesday, 13 March 2018 18:14
To: Beck, Zak <***@accenture.com<mailto:***@accenture.com>>; Xymon Mailing List <***@xymon.com<mailto:***@xymon.com>>
Subject: [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client

I have noticed a memory and thread leak in the Windows Virtual Disk service - or vds.exe from the process list. This service is set to Manual by default and gets started when starting the XymonPSClient service on Windows 2008 R2. It starts off using 32 threads, but grows forever, the virtual memory and physical memory (which starts under 6 MB) used slowly grow too.

Has anyone noticed this problem and do they have a solution to it?

On Windows Server 2012, the Virtual Disk service does not get started by Xymon Windows Powershell client, so it's obviously not started directly by the client, but by a request for information. I looked into known memory leaks for vds.exe and the one I found was not applicable to 2008 R2, only 2008 and Vista. The version of vds.exe we have is later than the version with the fix: 6.1.7601.17514 from 2010.

The quick workaround is simply to stop the Virtual Disk Service - but as soon as the PSclient has used it, it does not stop properly and terminates unexpectedly, so one needs to set it not to restart on crash, or stop it immediately after starting, or to Disabled. But it would be good to know why it is starting, not stopping, and leaking threads / memory!

Kind regards,

SebA

________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com<http://www.accenture.com>

[SebA]

Hi Zak,

Yes, I agree, a setting in a config file to enable or disable these checks
would probably be best. Thanks for the additional information.

Kind regards,

SebA

Post by Beck, Zak
Hi Seb
Thanks for confirming, that’s good news – I was fearing it may be the core
disk utilisation functions which would be a much larger headache!

Post by SebA
Is the XymonDiskPart information used for anything? I mean, can it be

used for anything without writing plug-ins?
The [diskpart] section is used in one of our datacentres where they use
clustering and the information is used primarily for reporting. So no, it
is not used for anything by default.
I think the best solution here is for me to add a setting in
xymonclient_config.xml allowing you to turn off the diskpart function for
specific servers. You can then remove your comment and continue to use
future updates.
Zak
*Sent:* Friday, 30 March 2018 14:04
*Subject:* Re: [External] Memory leak in Windows Virtual Disk service
caused by Xymon Windows Powershell client
Hi Zak,
Thanks for your swift reply and sorry for my own delay! It's good to know
you have such extensive use of the PS client without this issue.
I think you are probably right. These issue has only appeared on servers
using Windows Cluster Failover, where various iSCSI disks are clustered.
For the record, my workaround of disabling the Virtual Disk Service caused
no bad effects on the server, but it does prevent diskpart from running. I
set the service back to Manual, and it started as soon as I ran diskpart
from an elevated Command Prompt. Exiting diskpart causes the VDS service
and proces to stop.
When the Virtual Disk Service was set to manual, and started just by me
[diskpart]
diskpart:::Clustered Unknown
[diskpart]
When VDS is started by me restarting the XymonPSclient service and it
[diskpart]
diskpart:Disk 0:278 GB:Not Clustered
diskpart:Disk 1:1024 MB:Clustered Active
diskpart:Disk 2:25 GB:Clustered Active
diskpart:Disk 3:500 GB:Clustered Active
diskpart:Disk 4:300 GB:Clustered Inactive
diskpart:Disk 5:1024 MB:Clustered Inactive
diskpart:Disk 6:500 GB:Clustered Inactive
diskpart:Disk 7:200 GB:Clustered Inactive
diskpart:Disk 8:1024 MB:Clustered Inactive
diskpart:Disk 9:1024 MB:Clustered Inactive
diskpart:Disk 10:200 GB:Clustered Inactive
diskpart:Disk 11:200 GB:Clustered Inactive
diskpart:Disk 12:500 GB:Clustered Active
diskpart:Disk 13:5120 MB:Clustered Active
diskpart:Disk 14:500 GB:Clustered Inactive
“list disk”
For each disk returned, “select disk {n}” and then “detail disk”
for each disk (which does not leak memory) vds.exe uses about an extra MB
of RAM when running due to the PS client.
I have also now commented out the line you suggest (' $script:diskpartData
= XymonDiskPart ') and now when I restart the XymonPSclient service with
VDS set to Manual it does not get restarted and left running (or restarted
at all). So this looks like a better workaround than just disabling the
service, but I will need to continue to monitor that.
Is the XymonDiskPart information used for anything? I mean, can it be
used for anything without writing plug-ins?
We do not have memory leaks in powershell.exe on these servers (although I
think I may have seen it on another server), so the issue above is not
related to Alessandro's issue.
Thanks again for your help Zak, and I hope this knowledge is of benefit to
you or others too.
Kind regards,
SebA
Hi Seb
I’ve got hundreds of VMs running the Powershell client on Windows Server
2008R2, 2012R2 and 2016 and I don’t see this issue. I just checked a couple
of servers, vds.exe is between 2MB and 20MB and around 21 threads. Same
version of vds.exe too.
I’m wondering if function XymonDiskPart has something to do with it. This
only runs on slow scans (every 6 hours-ish by default). This runs
‘diskpart’ and then
“list disk”
For each disk returned, “select disk {n}” and then “detail disk”
Maybe you could try this from the command prompt on a server with vds
stopped and see if it restarts it. You could also try commenting out this
line (3875, towards the end) in the client and see what happens (insert a #
$script:diskpartData = XymonDiskPart
Alternatively, it could be the disk utilisation code which is using
Windows API calls in kernel32.dll – FindFirstVolume, FindNextVolume,
FindVolumeClose, GetVolumeInformation, GetVolumePathNamesForVolumeNameW,
GetDiskFreeSpaceEx and GetDriveType. The MSDN documentation for these
doesn’t mention the Virtual Disk Service but I guess they could be using it.
Zak
*Sent:* Tuesday, 13 March 2018 18:14
*Subject:* [External] Memory leak in Windows Virtual Disk service caused
by Xymon Windows Powershell client
I have noticed a memory and thread leak in the Windows Virtual Disk
service - or vds.exe from the process list. This service is set to Manual
by default and gets started when starting the XymonPSClient service on
Windows 2008 R2. It starts off using 32 threads, but grows forever, the
virtual memory and physical memory (which starts under 6 MB) used slowly
grow too.
Has anyone noticed this problem and do they have a solution to it?
On Windows Server 2012, the Virtual Disk service does not get started by
Xymon Windows Powershell client, so it's obviously not started directly by
the client, but by a request for information. I looked into known memory
leaks for vds.exe and the one I found was not applicable to 2008 R2, only
2008 and Vista. The version of vds.exe we have is later than the version
with the fix: 6.1.7601.17514 from 2010.
The quick workaround is simply to stop the Virtual Disk Service - but as
soon as the PSclient has used it, it does not stop properly and terminates
unexpectedly, so one needs to set it not to restart on crash, or stop it
immediately after starting, or to Disabled. But it would be good to know
why it is starting, not stopping, and leaking threads / memory!
Kind regards,
SebA
------------------------------
This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise confidential information. If you have
received it in error, please notify the sender immediately and delete the
original. Any other use of the e-mail by you is prohibited. Where allowed
by local law, electronic communications with Accenture and its affiliates,
including e-mail and instant messaging (including content), may be scanned
by our systems for the purposes of information security and assessment of
internal compliance with Accenture policy.
____________________________________________________________
__________________________
www.accenture.com