ftp upload large file problem

[belau]

can we have patience page for upload ?

[vikings]

i don't think so

[mmauch]

no, that is not possible.

[gugus]

I've got same problem with uploading files bigger than 10Mb: The bluecoat report an ICAP error message.

On the Antivirus log file (IWSS), I found this error message:

WARNING: X-TE trailers not found, ICAP client does NOT support trickling for this type of transaction


Do you know how to configure the bluecoat for fix this problem ?

[keving]

I don't think there is any way to fix an ICAP error on the ProxySG.

Perhaps you should check with Trend?

[vn_gr]

I have the same problem, with the Proxy SG.

The present of the error message
2006/06/16 08:41:58 GMT-05:00 <2048> WARNING: X-TE trailers not found, ICAP client does NOT support trickling for this type of transaction

Reason:

The X-TE trailers are used for trickling and large file handling. This message states that the ICAP client you're using (Bluecoat) does not have support for this.

The issue here is that there are almost no ICAP clients that support trickling. Even if IWSS can scan contents of large files, the ICAP bluecoat solution does not currently support this feature. Hence, IWSS with Bluecoat cannot perform large file handling and with CISCO is currently support.

Trend Micro say:

I open the case with Trend Micro and the support say if I active the proxy IWSS without ICAP and enable special handling for large files this do match for the rule. And IWSS with Bluecoat cannot perform large file handling and with CISCO is currently support.


I have 2 installations and before not sent the error in the log file the IWSS

Currently I try with the SGOS 4.1.2.12,---- 4.1.3.1,---- 4.1.4.1,---4.2.1.6
and the result is the same, I think that integration exist in some time, because BlueCoat change some.
Also I did the test with IWSS 2.0 AND 2.5 this very easy because Trend only work with 2 versions versus Bluecoat that work N versions the SGOS
and the resul is the same.

The problem is the BlueCoat!!!!!!

Basically, my explanation details as to how IWSS scans the packets. When handling large files, IWSS checks for a particular X-TE file handling ICAP header. This header is currently only supported for the NetCache appliance.
You know if Bluecoat now have or are working in some fix or new release???????

I expect your comments!!
Thank you

Vn

[keving]

I think support for trickling is coming (9-12 months perhaps).

Unfortunately IWSS is no longer 'officially supported' so I can't promise it will work with Trend.

[RedTrousers]

We also have issues with large file uploads where ICAP times out.

What we do is detect that it's a large upload, and if so forward over HTTP instead of ICAP. After that, the antivirus comes back to the bluecoat on another port and it goes to the internet.
It's all done in CPL, if someone's interested, I can post the code.

[stimpy99]

We also have issues with large file uploads where ICAP times out.

What we do is detect that it's a large upload, and if so forward over HTTP instead of ICAP. After that, the antivirus comes back to the bluecoat on another port and it goes to the internet.
It's all done in CPL, if someone's interested, I can post the code.

We are experiencing similar problems with BC and large Ftp uploads. A look at you work around would be nice. Thanks

[mi1tang4]

Hi RedTrousers,

Is it possible to have a look at your workaround ? We are having the file upload problem as well. Thanks.

[RedTrousers]

I've PM'd this to some others in the past, but since we're no longer using this (recent Finjan versions have preview enabled in request mode ICAP, thus solving the issue) I'll post it here for future reference.

Note that I've stripped all other code to protect the innocent. You may need to mingle this with your own local CPL if you have any.

I've also added some more comments to help clarify things

Here we go:

-----------------------------------------
; This is in the local CPL file by joost AT decock DOT org
; Conditions and actions are on top, the actual layers are at the bottom
;
; We have a Bluecoat (port 8082) that normally does ICAP to Finjan
; For big request (uploads over HTTP) we forward to Finjan over HTTP
; The Finjan's next proxy is once again the Bluecoat (port 8080)


; CONDITION: We run our proxy on port 8082
define condition CPL_Condition_ProxyPort_8082
proxy.port=8082
end

; CONDITION: Request larger then 3Mb (typically a file upload)
define condition CPL_Condition_Specific_RequestLargerThan3Mb
; Content-Length of 6 or more digits is way to big
request.header.Content-Length.length=6..
; Content-Length of 5 digits is to big if the first digit is 3-9
request.header.Content-Length.length=5 request.header.Content-Length="^[3-9]"
end


; CONDITION: Request larger then 3Mb proxied on port 8082
; This is just a combination of the two above
define condition CPL_Condition_GroupAND_LargeRequestProxyPort8082
condition=CPL_Condition_Specific_RequestLargerThan3Mb condition=CPL_Condition_ProxyPort_8082
end

; ACTION: Forward over HTTP, instead of using ICAP
define action CPL_Action_Specific_ForwardToFinjanBypassICAPStep1
; We add logging to have an idea of how frequent this happens
log_message("Big request detected. Not using ICAP, forwarding over HTTP. Rewriting headers.")
; In the forwarding layer, we have very limited context.
; Since the URL is one of the few things we have access to at that time, we are adding some stuff to trigger the forwarding
; We are adding '?BigPostWorkaround=on' at the end
rewrite(url, "(.*)", "$(1)?BigPostWorkaround=on")
; Also adding a X-BigPostAdded header
set(request.x_header.X-BigPostAdded, "yes")
; Strip the X-Bluecoat-Via header, or the Bluecoat will bark when the request returns
delete(request.x_header.X-Bluecoat-Via)
end

; Undo changes added by CPL_Action_Specific_ForwardToFinjanBypassICAPStep2
; When the request comes back after the forwarding, we re-normalize the request, undoing what we did before
define action CPL_Action_Specific_ForwardToFinjanBypassICAPStep2
; Some more informational logging
log_message("Forwarded big request received back from Finjan. Restoring headers.")
; Stripping our custom header
delete(request.x_header.X-BigPostAdded)
; Stripping the part we added to the URL
rewrite(url, "(.*)\?BigPostWorkaround=on", "$(1)")
end


<Proxy "CPL LAYER -- Bypass ICAP Request Mode">
; When the request first hits, this rule matches
; It triggers the action that adds headers and URL string
ALLOW condition=CPL_Condition_GroupAND_LargeRequestProxyPort8082 action.CPL_Action_Specific_ForwardToFinjanBypassICAPStep1(yes) client.protocol=http
; When it comes back from forwarding, this matches
; It triggers the action that undoes the modifications above
request.x_header.X-BigPostAdded.count=1 action.CPL_Action_Specific_ForwardToFinjanBypassICAPStep2(yes)

<Forward "CPL LAYER -- Forwarding">
; By the time this layer is treated, the URL has been modified
; We force the request through forwarding over HTTP instead of going over ICAP (the default in our case)
server_url.query.regex="BigPostWorkaround" proxy.port=8082 forward(finjan) forward.fail_open(no)

<Cache "CPL LAYER -- Bypass ICAP Response Mode">
; Finally, don't do response mode ICAP
server_url.query.regex="BigPostWorkaround" response.icap_service(no)
-----------------------------------------

That's it. Worked great for us, hopfully it can benefit others too.


ciao,

Joost