Note: Sorry, this is a long post. I realize off-network speeds are not guaranteed and there are many factors at play. I'm just looking to see if others are seeing the same issues and if anybody has any ideas on how to compensate for this or if this is something CV should be looking into.
I recently switched back to OOL from FiOS and noticed a pretty sharp decline in throughput to certain locations, especially west coast servers. I transfer quite a bit of data to/from San Jose and Los Angeles, and a fast upload speed from NY is important to me. With the upgrade to 101/35, I was finally ready to dump FiOS as the only reason I was sticking around was the 35Mbps upload.
However, it is impossible to reach anywhere close to 35Mbps. Most of the time, the connection seems capped at around 2-3Mbps:
mbp:Downloads td$ axel -n 1 -a "http://cablevisionIP/test1024"Initializing download: http://cablevisionIP/test1024File size: 1073752064 bytesOpening output file test1024Starting download [ 9%] [....0 ] [ 315.7KB/s] [50:15]
This is a transfer from my NY server to my laptop on Comcast in San Jose. If I open multiple connections, the transfer rate does improve, but it never reaches full speed. Heres a few examples with 6 and 24 connections:
mbp:Downloads td$ axel -n 6 -a "http://cablevisionIP/test1024"Initializing download: http://cablevisionIP/test1024File size: 1073752064 bytesOpening output file test1024Starting download [ 7%] [0 1 2 3 4 5 ] [ 1.2MB/s] [13:20]
mbp:Downloads td$ axel -n 24 -a "http://cablevisionIP/test1024"Initializing download: http://cablevisionIP/test1024File size: 1073752064 bytesOpening output file test1024Starting download [ 11%] [0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G ] [ 2.5MB/s] [06:07]
2.5MB/sec (20Mbps) is decent, but it is impractical for me to open 24 connections as most of my traffic is primarily over a VPN tunnel.
With a single connection to my server on FiOS (same server, settings, client laptop, etc)
mbp:Downloads td$ axel -n 1 -a "http://fiosIP/test1024"Initializing download: http://fiosIP/test1024File size: 1073752064 bytesOpening output file test1024Starting download [ 27%] [.............0 ] [ 3.2MB/s] [03:56]
With a single connection, it starts slow and fluctuates, but is pegged at 4.8MB/sec for most of the transfer. The entire transfer averages out to around 3.2-3.5MB/sec.
To compare the routing, this is a trace route from NY to San Jose:
CV to Comcast:
root@Zooropa:~# traceroute comcastIPtraceroute to comcastIP (comcastIP), 30 hops max, 60 byte packets 1 Gargoyle.home (192.168.1.1) 0.522 ms 0.453 ms 0.487 ms 2 10.240.161.57 (10.240.161.57) 7.677 ms 12.119 ms 13.053 ms 3 433be0d1.cst.lightpath.net (67.59.243.209) 13.051 ms 13.084 ms 13.131 ms 4 rtr2-ge1-6.mhe.whplny.cv.net (67.83.225.5) 14.022 ms 13.986 ms 14.065 ms 5 65.19.121.57 (65.19.121.57) 15.906 ms 15.870 ms 65.19.121.61 (65.19.121.61) 16.846 ms 6 64.15.3.109 (64.15.3.109) 14.892 ms 64.15.1.85 (64.15.1.85) 12.708 ms 64.15.3.105 (64.15.3.105) 14.626 ms 7 * * * 8 pos-1-11-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.86.189) 15.117 ms 16.390 ms 16.473 ms 9 he-0-0-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.88.130) 37.131 ms 40.491 ms 41.462 ms10 he-1-12-0-0-cr01.denver.co.ibone.comcast.net (68.86.85.250) 64.205 ms 64.156 ms 64.196 ms11 he-0-8-0-0-cr01.denverqwest.co.ibone.comcast.net (68.86.89.74) 67.564 ms 66.567 ms 67.509 ms12 he-4-10-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.89.117) 92.103 ms 91.199 ms 87.497 ms13 he-2-6-0-0-cr01.sacramento.ca.ibone.comcast.net (68.86.86.201) 89.116 ms 88.878 ms 88.721 ms14 he-0-7-0-0-ar01.oakland.ca.sfba.comcast.net (68.86.90.130) 87.316 ms 88.672 ms 86.737 ms15 68.85.155.53 (68.85.155.53) 87.767 ms 88.652 ms 91.118 ms16 te-17-10-cdn39.sanjose.ca.sfba.comcast.net (68.87.195.58) 89.889 ms 103.315 ms 88.850 ms17 comcastIP.ca.comcast.net (comcastIP) 96.573 ms 105.175 ms 96.003 ms
FiOS to Comcast:
root@Zooropa:~# traceroute -i eth1 comcastIPtraceroute to comcastIP (comcastIP), 30 hops max, 60 byte packets 1 Gargoyle.home (192.168.1.1) 0.382 ms 0.480 ms 0.513 ms 2 L100.NYCMNY-VFTTP-169.verizon-gni.net (71.190.135.1) 2.796 ms 2.803 ms 3.654 ms 3 G0-13-1-5.NYCMNY-LCR-22.verizon-gni.net (130.81.182.78) 7.559 ms 7.579 ms 7.549 ms 4 so-6-1-0-0.NY5030-BB-RTR2.verizon-gni.net (130.81.151.224) 8.094 ms 8.097 ms 9.394 ms 5 0.xe-7-3-1.XL4.NYC1.ALTER.NET (152.63.10.205) 8.931 ms 10.317 ms 10.315 ms 6 0.xe-9-1-0.GW13.NYC1.ALTER.NET (152.63.4.141) 11.551 ms 9.968 ms 9.889 ms 7 comcast.com.customer.alter.net (152.179.220.126) 12.988 ms 5.666 ms 6.380 ms 8 he-0-4-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.83.93) 8.785 ms 8.754 ms 7.792 ms 9 he-0-1-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.88.134) 26.943 ms 26.921 ms 26.932 ms10 he-1-9-0-0-cr01.denver.co.ibone.comcast.net (68.86.89.42) 58.962 ms 59.040 ms 59.041 ms11 he-0-2-0-0-cr01.denverqwest.co.ibone.comcast.net (68.86.89.30) 67.037 ms 67.039 ms 67.036 ms12 he-4-13-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.89.129) 76.359 ms 76.251 ms 76.244 ms13 he-2-0-0-0-cr01.sacramento.ca.ibone.comcast.net (68.86.85.49) 73.461 ms 72.045 ms 72.330 ms14 he-0-6-0-0-ar01.oakland.ca.sfba.comcast.net (68.86.91.226) 78.140 ms 75.131 ms 75.107 ms15 68.85.155.49 (68.85.155.49) 75.085 ms 75.125 ms 75.113 ms16 te-17-10-cdn39.sanjose.ca.sfba.comcast.net (68.87.195.58) 74.348 ms 74.983 ms 89.024 ms17 comcastIP.hsd1.ca.comcast.net (comcastIP) 84.470 ms 84.431 ms 84.412 ms
So you can see after hop 8, both travel the same exact path. It could be that Comcast just has better peering with Verizon than CableVision, but here are some other tests with other networks:
[att=1]
As an example, heres the trace route to the Chicago server from both:
CV to TowerStream (Chicago)
root@Zooropa:~# traceroute 69.38.192.34traceroute to 69.38.192.34 (69.38.192.34), 30 hops max, 60 byte packets 1 Gargoyle.home (192.168.1.1) 0.459 ms 0.514 ms 0.545 ms 2 10.240.161.57 (10.240.161.57) 8.008 ms 12.474 ms 13.255 ms 3 67.59.228.113 (67.59.228.113) 13.288 ms 13.284 ms 13.311 ms 4 rtr1-ge2-6.mhe.whplny.cv.net (67.83.225.9) 14.244 ms 14.312 ms 14.389 ms 5 65.19.121.57 (65.19.121.57) 14.165 ms 14.196 ms 14.188 ms 6 64.15.1.22 (64.15.1.22) 15.092 ms rtr3-tg11-1.in.nycmny83.cv.net (64.15.0.53) 13.602 ms 13.476 ms 7 64.15.0.118 (64.15.0.118) 36.116 ms 175.289 ms 79.386 ms 8 xe-6-0-3.ar1.ord1.us.nlayer.net (69.31.111.169) 45.064 ms 40.734 ms 41.546 ms 9 as32654.xe-10-0-6.ar1.ord1.us.nlayer.net (69.31.110.66) 39.755 ms 38.624 ms 37.564 ms10 69.38.199.66 (69.38.199.66) 38.559 ms 38.560 ms 38.636 ms11 g1-0-2.dr.chi2.ill.towerstream.net (69.38.194.30) 46.939 ms 52.301 ms 53.268 ms
FiOS to TowerStream (Chicago)
root@Zooropa:~# traceroute 69.38.192.34 -i eth1traceroute to 69.38.192.34 (69.38.192.34), 30 hops max, 60 byte packets 1 Gargoyle.home (192.168.1.1) 0.385 ms 0.450 ms 0.489 ms 2 L100.NYCMNY-VFTTP-169.verizon-gni.net (71.190.135.1) 1.728 ms 1.798 ms 1.855 ms 3 G0-13-1-5.NYCMNY-LCR-22.verizon-gni.net (130.81.182.78) 3.713 ms 3.709 ms 3.697 ms 4 ae0-0.NY5030-BB-RTR1.verizon-gni.net (130.81.209.118) 3.894 ms 3.939 ms 3.916 ms 5 0.xe-11-3-0.BR2.NYC4.ALTER.NET (152.63.23.137) 4.306 ms 4.257 ms 4.227 ms 6 204.255.168.110 (204.255.168.110) 85.052 ms 83.400 ms 83.444 ms 7 te0-2-1-2.ccr21.jfk02.atlas.cogentco.com (154.54.5.209) 89.565 ms 89.571 ms 82.860 ms 8 te0-7-0-15.ccr21.ord01.atlas.cogentco.com (154.54.43.90) 53.039 ms 51.421 ms * 9 te3-7.mag01.ord01.atlas.cogentco.com (154.54.45.194) 52.956 ms 52.893 ms 53.927 ms10 vl3616.na21.b000619-0.ord01.atlas.cogentco.com (38.20.41.226) 54.064 ms 55.780 ms 55.714 ms11 cogent.chi.il.towerstream.com (38.104.98.182) 49.591 ms 49.547 ms 50.495 ms12 g1-0-2.dr.chi2.ill.towerstream.net (69.38.194.30) 52.348 ms 52.340 ms 53.290 ms
Obviously Verizon uses cogent while CV uses a different peer for this particular server, so its not a direct comparison. For the Denver server, both use the same peer (twtelecom):
CV to Denver
root@Zooropa:~# traceroute 199.88.135.249traceroute to 199.88.135.249 (199.88.135.249), 30 hops max, 60 byte packets 1 Gargoyle.home (192.168.1.1) 0.361 ms 0.420 ms 0.471 ms 2 10.240.161.57 (10.240.161.57) 9.599 ms 10.485 ms 10.485 ms 3 67.59.228.109 (67.59.228.109) 10.543 ms 11.446 ms 11.457 ms 4 rtr1-ge2-6.mhe.whplny.cv.net (67.83.225.9) 12.372 ms 14.888 ms 15.119 ms 5 65.19.121.61 (65.19.121.61) 16.304 ms 16.351 ms 65.19.121.53 (65.19.121.53) 17.115 ms 6 451be0a9.cst.lightpath.net (65.19.120.169) 16.339 ms 64.15.3.121 (64.15.3.121) 12.655 ms 64.15.3.117 (64.15.3.117) 12.683 ms 7 64.15.3.157 (64.15.3.157) 18.916 ms 17.818 ms 16.982 ms 8 * * * 9 den1-ar4-xe-2-1-0-0.us.twtelecom.net (206.222.119.66) 65.745 ms 64.811 ms 65.750 ms10 66.195.94.202 (66.195.94.202) 66.680 ms 66.638 ms 66.674 ms11 199.88.135.249 (199.88.135.249) 66.628 ms 66.642 ms 66.665 ms
FiOS to Denver
root@Zooropa:~# traceroute 199.88.135.249 -i eth1traceroute to 199.88.135.249 (199.88.135.249), 30 hops max, 60 byte packets 1 Gargoyle.home (192.168.1.1) 0.421 ms 0.348 ms 0.394 ms 2 L100.NYCMNY-VFTTP-169.verizon-gni.net (71.190.135.1) 2.174 ms 2.237 ms 3.195 ms 3 G0-13-3-0.NYCMNY-LCR-21.verizon-gni.net (130.81.183.242) 6.770 ms 6.690 ms 6.627 ms 4 130.81.199.10 (130.81.199.10) 7.624 ms 7.579 ms 8.645 ms 5 0.ae2.XL3.DFW7.ALTER.NET (140.222.226.103) 53.724 ms 53.720 ms 53.662 ms 6 TenGigE0-6-0-0.GW6.DFW13.ALTER.NET (152.63.97.102) 47.793 ms 47.015 ms 46.975 ms 7 twtelecom-gw.customer.alter.net (152.179.51.234) 43.688 ms 42.393 ms 42.364 ms 8 den1-ar4-xe-2-1-0-0.us.twtelecom.net (206.222.119.66) 58.874 ms 56.820 ms 56.686 ms 9 66.195.94.202 (66.195.94.202) 57.635 ms 57.270 ms 57.395 ms10 199.88.135.249 (199.88.135.249) 56.065 ms 55.816 ms 55.953 ms
As I said at the beginning, I'm just looking for any input on how to get around this limitation. I've tried using a Hurricane Electric IPv6 tunnel, but CV's peering with HE is pretty horrendous as well. I tried setting up an Amazon EC2 virtual server in Virginia, but I can't break 8Mbps upload to even Amazon's server. What are the odds of CV actually doing something about this? Even if they would, who should I contact? The regular CSRs just tell me that they can't guarantee off-network speeds.
--
University of Southern California - Fight On!
↧