Devices downloading large amounts of data - how to identify what this is

Network monitoring tools show large amounts of data being sent from ZCM primary or satellite servers to devices, often flooding bandwidth.

How can administrators identify what this traffic is?

Preparation

• Take a network trace from the ZCM server sending the data
• Set up a packet capture tool such as tcpdump or Wireshark with a circular buffer so that traffic is already being captured when the problem occurs
  
• From network management tools identify a sample of devices that are receiving large amounts of data

Analysis

• The following example assumes that a device with an address of 10.20.2.134 has downloaded 60Mb and that Wireshark is the packet analyser being used
• Find the data stream with the following filter
  http.request.method == "GET" && ip.addr==10.20.2.134
• Which shows a packet similar to the following
  10.20.2.134    50777    10.18.4.77    80    HTTP    GET /zenworks-content/ed62231c73d3e917e4f0003608821b23.zc HTTP/1.1
• Right-click this and select Follow TCP Stream which should show something similar to the following
  GET /zenworks-content/ed62231c73d3e917e4f0003608821b23.zc HTTP/1.1
  Request-Priority: 100
  Host: 10.18.4.77
  
  HTTP/1.1 200 OK
  Server: Apache-Coyote/1.1
  Last-Modified: Wed, 14 Nov 2012 18:01:13 GMT
  Content-Type: application/octet-stream
  Content-Length: 63326116
  Date: Tue, 20 Nov 2012 09:36:16 GMT
• Take the GUID and run the following query in the database
  select * from zContentFileNames where id=0xed62231c73d3e917e4f0003608821b23;
• Which should then return the filename being downloaded
  id                                   FileNames                      position
  0xed62231c73d3e917e4f0003608821b23   windowsxp-bigapp-x86-enu.exe   0
  
• From there, check the schedule and whether it is a valid bundle
• To see which other devices are also downloading the same bundle, use the following Wireshark filter
  http.request.method == "GET" && http.request.uri == "/zenworks-content/ed62231c73d3e917e4f0003608821b23.zc"
  

In one customer case study, large amounts of data were being sent from the primary servers across the WAN to devices on another site, using up all the WAN bandwidth.  This procedure was used to identify the fact that a Patch DAU update was simultaneously being sent to over 1000 devices.  The reason was that a local Satellite server had run out of disk space and was unable to cache the new bundle so devices were crossing the WAN back to a primary to retrieve the necessary updates.