前几天在做一个使用URL通过WebRequest请求HTML页面的功能的时候遇到了点坑，程序在开发环境没有任何的问题，部署到linux mono上之后就跪了。代码如下：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
| public static string GetHTML(string url)
{
string htmlCode;
try
{
HttpWebRequest webRequest = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(url);
webRequest.Timeout = 30000;
webRequest.Method = "GET";
webRequest.UserAgent = "Mozilla/4.0";
webRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
HttpWebResponse webResponse = (System.Net.HttpWebResponse)webRequest.GetResponse();
//获取目标网站的编码格式
string contentype = webResponse.Headers["Content-Type"];
Regex regex = new Regex("charset\\s*=\\s*[\\W]?\\s*([\\w-]+)", RegexOptions.IgnoreCase);
if (webResponse.ContentEncoding.ToLower() == "gzip")//如果使用了GZip则先解压
{
using (System.IO.Stream streamReceive = webResponse.GetResponseStream())
{
using (var zipStream = new System.IO.Compression.GZipStream(streamReceive,
System.IO.Compression.CompressionMode.Decompress))
{
//匹配编码格式
if (regex.IsMatch(contentype))
{
Encoding ending = Encoding.GetEncoding(regex.Match(contentype).Groups[1].Value.Trim());
using (StreamReader sr = new System.IO.StreamReader(zipStream, ending))
{
htmlCode = sr.ReadToEnd();
}
}
else
{
using (StreamReader sr =
new System.IO.StreamReader(zipStream, Encoding.UTF8))
{
htmlCode = sr.ReadToEnd();
}
}
}
}
}
else
{
using (System.IO.Stream streamReceive = webResponse.GetResponseStream())
{
using (System.IO.StreamReader sr = new System.IO.StreamReader(streamReceive, Encoding.Default))
{
htmlCode = sr.ReadToEnd();
}
}
}
return htmlCode;
}catch(Exception ex)
{
LogHelper.WriteException("GetHTML", ex, new { Url = url });
return "";
}
}
|
开发环境在Windows10 + VS2013,整个代码跑起来没什么问题。
无论是HTTP还是HTTPS协议，网页HTML一样能获取得到。
网站部署到linux Jexus之后HTTP协议的网站同样可以获取到HTML，遇到HTTPS协议的网站的时候就跪了。
抓到的异常信息如下：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
| System.Net.WebException: Error: TrustFailure (The authentication or decryption has failed.)
---> System.IO.IOException: The authentication or decryption has failed.
---> System.IO.IOException: The authentication or decryption has failed.
---> Mono.Security.Protocol.Tls.TlsException:
Invalid certificate received from server. Error code: 0xffffffff800b0109
at Mono.Security.Protocol.Tls.RecordProtocol.EndReceiveRecord
(IAsyncResult asyncResult) <0x41b58d80 + 0x0013e> in <filename unknown>:0
at Mono.Security.Protocol.Tls.SslClientStream.SafeEndReceiveRecord
(IAsyncResult ar, Boolean ignoreEmpty) <0x41b58cb0 + 0x00031> in <filename unknown>:0
at Mono.Security.Protocol.Tls.SslClientStream.NegotiateAsyncWorker
(IAsyncResult result) <0x41b72a40 + 0x0023f> in <filename unknown>:0
--- End of inner exception stack trace ---
at Mono.Security.Protocol.Tls.SslClientStream.EndNegotiateHandshake
(IAsyncResult result) <0x41ba07e0 + 0x000f3> in <filename unknown>:0
at Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback
(IAsyncResult asyncResult) <0x41ba0540 + 0x00086> in <filename unknown>:0
--- End of inner exception stack trace ---
at Mono.Security.Protocol.Tls.SslStreamBase.EndRead
(IAsyncResult asyncResult) <0x41b73fd0 + 0x00199> in <filename unknown>:0
at Mono.Net.Security.Private.LegacySslStream.EndAuthenticateAsClient
(IAsyncResult asyncResult) <0x41b73f30 + 0x00042> in <filename unknown>:0
at Mono.Net.Security.Private.LegacySslStream.AuthenticateAsClient (System.String targetHost,
System.Security.Cryptography.X509Certificates.X509CertificateCollection
clientCertificates, SslProtocols enabledSslProtocols,
Boolean checkCertificateRevocation) <0x41b6a660 + 0x00055> in <filename unknown>:0
at Mono.Net.Security.MonoTlsStream.CreateStream (System.Byte[] buffer)
<0x41b69c30 + 0x00159> in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult)
<0x41b67660 + 0x001f9> in <filename unknown>:0
at System.Net.HttpWebRequest.GetResponse () <0x41b60920 + 0x0005a> in <filename unknown>:0
at WebBookmarkUI.Commom.HTTPHelper.GetHTML (System.String url) <0x41b59b70 + 0x00235>
in <filename unknown>:0
|
有空的信息基本就是：
- Invalid certificate received from server
- The authentication or decryption has failed
一开始百思不得其解，为嘛好好的程序在开发环境跑得都好的，到了mono上就挂了，多疑的我还以为是mono的bug。
今天静下心来去找了一下资料，发现Mono的文档有这个问题的描述，认真读了一遍，又去请教了一番宇内流云大大，终于弄懂了是什么回事。
先贴一下相关资料：
stackoverflow mono-webrequest-fails-with-https
mono doc security
这个问题是出现的原因是Windows自带了HTPPS的根证书，linux默认则是没有带有根证书的。我们的mono在调用WebRequest去请求HTTPS协议的网站的时候，抛出上上面的异常了。
解决方案也很简单，为linux导入一下HTTPS根证书就好。
在linux服务器上面执行下面这条命令。
1
| mozroots --import --ask-remove --machine
|
然后在网站的Application_Start()里面添加下面代码：
1
2
3
4
5
6
7
| System.Net.ServicePointManager.ServerCertificateValidationCallback +=
delegate(object sender, System.Security.Cryptography.X509Certificates.X509Certificate certificate,
System.Security.Cryptography.X509Certificates.X509Chain chain,
System.Net.Security.SslPolicyErrors sslPolicyErrors)
{
return true; // **** Always accept
};
|
完事。
这个故事告诉我们，linux干活都是要亲力亲为呀。