Topic: Proxy slows down fast connection

[Proxy]

Hi there,

I have a fast internet connection 100Mbit up and download. (Optical Fiber)
That's about 12MByte.
Works fine, till i put on the Proxy, then speeds slows down to 4-4,5Mbyte
Can anyone tell me why?
Is there a setting in the proxy to solve this problem?

I like to hear from you

Best regards,
Proxy

[cactus]

I have a fast internet connection 100Mbit up and download. (Optical Fiber)
That's about 12MByte.

Theoretically yes, but in practice you ill not see that most of the times as part of the data transmission bandwidth is used for sending checksums and doing error corrections in the data stream.

Works fine, till i put on the Proxy, then speeds slows down to 4-4,5Mbyte
Can anyone tell me why?
Is there a setting in the proxy to solve this problem?

AFAIK speed might be lost due to data analysis but I am not sure it would be that much. AFAIK there is no parameter to work your proxy work faster. It might also depend on hardware the proxy is using an older CPU might have to little procession speed to or seamlessly with the large amount of data you are requesting. It might also be that the upstream provider is not allowing you to use the full 100Mbit bandwidth and it is throttling the connection. You are not giving us enough information to go on.

[Proxy]

Oke here comes more info.

I am using SMEServer v7.4 normal installation.
CPU 800Mhz (Little computer)
512MB Internal memory
20GB Harddisk
1Gbit router D-Link dir-655
100Mbit networkadapter at server and pc
5 PC's are attachted to the server.

The providor is not throttling the connection. it's 100Mbit up and 100Mbit down duplex.
(approximately 1GB download in 00:01:30 seconds)
Maby it's the hardware but... with proxy off it works fine, i get up and downloadspeed to 11,2 - 11,4 MByte.

Do you know if the proxy can handle duplex connections, perhapps that is the problem?

[Stefano]

CPU 800Mhz (Little computer)
512MB Internal memory
20GB Harddisk
1Gbit router D-Link dir-655
100Mbit networkadapter at server and pc
5 PC's are attachted to the server.

you should use a faster cpu if you want to handle such a connection

Ciao
Stefano

[Proxy]

So you think with proxy ON the CPU is to light.
And with proxy OFF it's oke
Hmm,.. maybe possible.. because with proxy ON the load is to much for the poor CPU.

[electroman00]

1Gbit router D-Link dir-655
100Mbit networkadapter at server and pc

Are there any servers or pc's using 1000T on the 655 lan ports?????????????

The 655 is really not made to have 10T, 100T and 1000T commingled at the same time, 655 doesn't have buffers required to handle
different rates at the same time efficiently.

Although it will work, performance will be noticeably degraded.

Proxy is a mundane task for SME so 800mhz isn't going to be the issue.

part of the data transmission bandwidth is used for sending checksums and doing error corrections in the data stream.

part of the data transmission bandwidth is used for packet checksums and error trapping in the data stream.

The bandwidth loss due to that, will only be an issue if there are packet issues.

Packets are checked for integrity, if trapped bad, they are dropped and a request sent to resend the packet.

With all that said, the problem is likely to be because you might be using Server-Only Mode.

If you are, then that is where your loosing bandwidth performance with a proxy.

Why, because you can't proxy to SME's Lan interface in server-only mode (for obvious reasons), you must create a remote proxy.

When you remote proxy, there are two connections thru the same external interface, that share the bandwidth thru that interface.

The external connection and the proxy connection.

When you proxy to SME's LAN (gateway mode) only one connection is made thru SME's external interface, thus no bandwidth sharing thru the external interface.

Server-Only mode you are bandwidth sharing all connections made thru SME's external interface, whether you want to or not.

hth

That's one of the reasons why Gateway mode is the preferred SME mode.

SME in Gateway mode will run on any network providing you set up the network properly.

I have a fast internet connection 100Mbit up and download. (Optical Fiber)
That's about 12MByte.
Works fine, till i put on the Proxy, then speeds slows down to 4-4,5Mbyte

Not byte....bit.

Definitely some confusion there.

Not likely your ISP will give you the same U/D bandwidth with a residential connection.

I assume your FIOS, since you did not mention.

FiOS Internet Service is the fastest and most powerful connection we've ever offered. With FiOS, you have the choice of several plans to meet your needs. We offer downstream connection speeds (the speed that data is transmitted to your computer) ranging from up to 10 Mbps all the way up to 50 Mbps. We offer upstream speeds up to 20 Mbps in some locations.

If you have their "Faster Plus" plan then your U/D bandwidth is matched 20Mbit not byte.

The "Fastest" plan is not matched U/D.

hth

[electroman00]

I might add.....

Works fine, till i put on the Proxy, then speeds slows down to 4-4,5Mbit

Thats about right for server-only mode.

[electroman00]

Also goto speedtest.net and you will see your U/D bandwidth.

Large gauge is max bandwidth, small gauge is flow.

hth

[electroman00]

Also again....

Although a 100T nic does have more then enough bandwidth to handle two 12mbit connections,
some of the older 100T nic's simply don't perform very well.

And of course you have to consider all other simultaneous traffic thru the network.

If you suspect a problem, it's a good idea to do some diag testing, ping, tracert, mtu ping testing
which might shed some light on any problems that may be degrading your results.

hth

[Proxy]

Oke, server is in Gateway-mode not server-only.
Internetconnection is 100 Mbps full duplex. that is about 12,5Mbyte (:
With proxy ON speeds drop from 12,5Mbyte (or 11,4 what ever you like)
to 4-4,5Mbyte - not Mbits!
Workstation had nothing to do with it becaurse with proxy OFF it's works fine.

[cactus]

I have a fast internet connection 100Mbit up and download. (Optical Fiber)
That's about 12MByte.
Works fine, till i put on the Proxy, then speeds slows down to 4-4,5Mbyte

Not byte....bit.

Nope, in this case the user is right:

100Mbit = 12,5 Mbyte as 1 byte is 8 bits.

[mmccarn]

you should use a faster cpu if you want to handle such a connection

I second neonano's recommendation - try a faster computer for your SME if you need to use the proxy.

My understanding is that the main performance enhancement provided by squid involves keeping local cached copies of thousands of small files.  My home SME, for example, has 20,000 files in /var/spool/squid.

Your proxy server needs to be able to perform high speed calculations (to locate the desired file), high speed disk I/O (to locate and read the requested file) and high speed network communications (to get new cache content and serve that content to your LAN).

So - while I wouldn't be surprised if your 800MHz system could serve websites at 12.5MBytes/sec, it also doesn't surprise me that it doesn't...

[CharlieBrady]

Works fine, till i put on the Proxy, then speeds slows down to 4-4,5Mbyte

How are you measuring that speed? Use of the proxy introduces additional latency,  since the proxy must interpret the request, then search its cache, then send the request upstream and start storing to disk, then start sending the file stream to the client. That takes time. Streaming should be just about as fast once it starts happening.

[Proxy]

Speed is measured at website below (onsbrabantnet) i download the file 1024MB
OnsBrabantNet is the provider
http://www.glasvezel.nu/speedtest-servers-100mb-1000mb-bin
The file is not stored in cache (i store max 64MB)

"try a faster computer for your SME if you need to use the proxy."
Yes i think also this is the problem, however CPU speed, load etc... are normal, measured by Sysmon.

[electroman00]

Not byte....bit.
Nope, in this case the user is right:

100Mbit = 12,5 Mbyte as 1 byte is 8 bits.

Thanks cactus my friend, but you weren't the fish I was trying to catch.

Definitely some confusion there.

Where???
Not likely you will actually get 12.5MBytes with a 100Mbit NIC.
So is the number wrong or the multiplier wrong?

(or 11.4 what ever you like)

Appears the number.
11.4 ......I really do like.
BTW thanks for the multiplier verification also.

For the astute....

Thats about right for server-only mode.

With 11.4 the calc now works precisely, well as precisely as the measurements and numbers presented.

2 connections thru SME's external interface 100Mbit NIC.

Max 100T NIC is 100Mbit = 12.5MByte

Which equals
12.5Mbyte / 2 = 6.25MByte
So the best you could possibly get is 6.25Mbyte (if the sun is shining) and thats if you don't consider any thing else in the real networking world.

So let's correct it for the newly/recently presented real data of 11.4
11.4 / 2 = 5.7MBytes
Subtract some for the connection to the test server and some for the usual network comm stuff.
Let's say for round numbers 0.7Mbyte.....we now get
5.7 - 0.7 = 5.0 MByte

Hummmm 5.0 MByte

The reported speeds slows down to 4-4,5Mbyte.

I'd say.....Snoopy, were in the ball ParK + / - a few bitty Bytes.

Oh that's right, Proxy said he is using Gateway Mode, so all this is nonsense.

Sorry...my bad.

Interesting, you learn something new every day.

Everyone here, is saying the CPU is the predominate bandwidth controlling hardware device in a computer.

To think, all these years I thought it was the Ethernet Network Interface Card.

So if I by a Cray computer and install a 100Mbit nic, then, the bandwidth problem will be fixed.

Man, I wish I would have known this, we wasted all that money on a new gigabit network.
And we made all those customers upgrade, boy are they going to be mad when they hear this.

I just don't know how such a simple concept could have eluded me for so long.

To think I listened to all the people I thought really knew something.

Cactus your right....
You have to "Be careful whose advice you buy,".....very careful.

Oh well such is life, you live and learn.

Proxy

In Server-Manager under "Hostnames and addresses" what did you select for Location?

hok

[Proxy]

Nothing i get announcement:

Operationeel status rapport
ect....

"Which equals
12.5Mbyte / 2 = 6.25MByte
So the best you could possibly get is 6.25Mbyte (if the sun is shining) and thats if you don't consider any thing else in the real networking world."

Nice thinking but with proxy OFF,.. i get the full speed of 11,4MByte.

[electroman00]

Nothing i get announcement:

Operationeel status rapport
ect....

"Which equals
12.5Mbyte / 2 = 6.25MByte
So the best you could possibly get is 6.25Mbyte (if the sun is shining) and thats if you don't consider any thing else in the real networking world."

Nice thinking but with proxy OFF,.. i get the full speed of 11,4MByte.

Again you confirmed the theory.

So the proxy you setup points to a server I assume on SME's Lan address space, is that correct.

[mmccarn]

...I have a fast internet connection 100Mbit up and download. (Optical Fiber)

According to the pfSense Hardware Sizing Guidance:

Throughput Considerations

If you require less than 10 Mbps of throughput, you can get by with the minimum requirements. For higher throughput requirements we recommend following these guidelines, based on our extensive testing and deployment experience. These guidelines offer a bit of breathing room because you never want to run your hardware to its full capacity.

10-20 Mbps - No less than 266 MHz CPU
21-50 Mbps - No less than 500 MHz CPU
51-200 Mbps - No less than 1.0 GHz CPU
201-500 Mbps - server class hardware with PCI-X or PCI-e network adapters, or newer desktop hardware with PCI-e network adapters. No less than 2.0 GHz CPU.
501+ Mbps - server class hardware with PCI-X or PCI-e network adapters. No less than 3.0 GHz CPU.

which I read to say "get a faster CPU" for your 100Mbps connection... 

[Normando]

Nice thinking but with proxy OFF,.. i get the full speed of 11,4MByte.

You must use a faster CPU with a faster HD. This is the problem in your server. Your slow CPU and slow HD is a bottleneck, so in your server is better switching off the proxy, or upgrade the hardware.

[Proxy]

Yes i think to that a faster CPU and HD solves the problem.
Thanks all for your advice!

[electroman00]

According to the pfSense Hardware Sizing Guidance:
which I read to say "get a faster CPU" for your 100Mbps connection... 

The above chart considers multiple connections, this issue involves only 2 plus whatever is current at time of testing.

At the CL enter...
top or htop
That will help you determine if your CPU is taxed or overloaded.

iptraf
Will allow you to monitor eth1 traffic and will show the number of  simultaneous connections thru eth1.
Don't be surprised if the list gets long.
You can monitor all traffic thru both eth0 & eth1 at the same time also.

You can open two shells and run both top and iptraf at the same time.
Should give you a clear picture as to where the problem lies.

Linux has many tools for diagnosing problems, always a good idea to become familiar with them and the info they provide.

Scenario 1 Gateway Mode
Source Request > SME WEB IP (via SME's external interface 100Mbit NIC connection 1) >>> Proxy SME's Gateway IP (via SME's internal interface 100Mbit??? NIC connection 2)

1 connection thru SME's external interface 100Mbit NIC.
Max Bandwidth for the 1 connection thru that interface = 12.5MByte or there a bouts.

Scenario 2 Gateway or Server Only Mode
Source Request > SME WEB IP (via SME's external  interface 100Mbit connection 1) >>> to Proxy Remote IP (via SME's  EXTERNAL 100Mbit connection 2)

2 connections thru SME's external interface 100Mbit NIC.

Max 100T NIC is 100Mbit = 12.5MByte

Which equals
12.5Mbyte / 2 = 6.25MByte
So the best you could possibly get is 6.25Mbyte and that's if you don't consider any thing else in the real networking world.

So let's correct it for the newly/recently presented real data of 11.4
11.4 / 2 = 5.7MBytes
Subtract some for the connection to the test server and some for the usual network comm stuff.
Let's say for round numbers 0.7Mbyte.....we now get....
5.7 - 0.7 = 5.0 MByte

5.0 MByte

The reported speeds slows down to 4-4,5Mbyte.

I'd say, your in the ball ParK +/- a few bitty Bytes.

Works fine, till i put on the Proxy, then speeds slows down to 4-4,5Mbit

Quote

That's about right for server-only mode.

It will also be right in Gateway mode IF you proxy off the server to a remote IP addy.
1 inbound connection for Wan and 1 outbound connection to the remote IP thru the external interface nic.

2 connections thru the external interface nic.
Thus the bandwidth is approx 11.4 / 2 = 5.7MBytes

or

Gateway mode proxy to local gateway IP.
1 inbound connection for Wan and local proxy (routed to GW interface) no outbound external connection to the remote IP.

1 connections thru the external interface nic.
Thus the bandwidth is approx 11.4MBytes or max bandwidth available at the current time.

Surely it doesn't hurt to upgrade the system to a faster CPU to sustain additional proxy connections, however
the bottleneck will still be the 100Mbit nic if that is not upgraded.

My experience is years back 400MHZ CPU and 1000Mbit nic will handle 12-16 connections fairly well, the performance then
becomes a function of the Wan bandwidth and speed of the Harddisk.

Since you already have a gigabit router, a gigabit nic will be a lot less then a new system for sure.
Intel gigabit nic works very well with SME, the Intel server motherboard we use have daul Intel nic's built in.

At any one given time the server here has 10-80 connections, at this very moment 13 connections thru eth1 open and 1 passing constant packets
due to a remote shell connection.

If you place a large file i.e. an iso on your remote server and setup multiple downloads simultaneously you can monitor what is happening.

Understanding is in the mind of the beholder. YMMV

HTH

[mercyh]

Why would you wish to proxy such a fast connection?

Generally proxy is used to speed up a slow connection, limit total bandwidth usage (by caching redundant downloads) or for filtering purposes.

In your case it doesn't look like you need the first two.

[Proxy]

Ooh, i was just testing and wonder why....
Still i can't check the many answers, i have at this moment no fast server available.

[mercyh]

Ooh, i was just testing and wonder why....
Still i can't check the many answers, i have at this moment no fast server available.

Fine,
I was just trying to determine what use a proxy would have on such a connection.

(I think most of us would be a bit jealous of your bandwidth)

[Proxy]

Yes i have lucky, in our street they just made fiberglass possible.
I pay about 45,- euro a month for it.
It's real fast - 1GB - in 00:01:30 seconds (at full speed)
i now have set up a Mirror for the SME Server (ftp://ftp.rgmtechnical.nl/)
i hope the FTP is save

[cactus]

Yes i have lucky, in our street they just made fiberglass possible.
I pay about 45,- euro a month for it.
It's real fast - 1GB - in 00:01:30 seconds (at full speed)
i now have set up a Mirror for the SME Server (ftp://ftp.rgmtechnical.nl/)
i hope the FTP is save

Wow, living in the proper part of Holland (at least for fibre)... I am and perhaps need to wait a long time for the glass to reach my doorstep.

Did you report a bug in the bugtracker specifying the details for your mirror so it can be monitored and when deamed stable added to the list of mirrors?

[Proxy]

You live in Maastricht.. Hmm i think in 2025 you have fiber 
Perhaps you can call reggefiber and ask them when Maastricht have fiber.

Mirror: The FTP site is mirrored and i am on the list, ftp://ftp.rgmtechnical.nl/mirrorlist/index.html

[cactus]

You live in Maastricht.. Hmm i think in 2025 you have fiber 
Perhaps you can call reggefiber and ask them when Maastricht have fiber.

So that means there is fibre outside of Amsterdam as well... we are slowly getting somewhere in our small country with (one of) the highest densities of internet connections. Perhaps I should have stayed in Twente, if only someone had a proper job for me back then

Mirror: The FTP site is mirrored and i am on the list, ftp://ftp.rgmtechnical.nl/mirrorlist/index.html

I know but the proper channel to announce your mirror and have it taken up in the monitor system is to open a bug for it in the bugtracker, so please do so.

[Proxy]

"I know but the proper channel to announce your mirror and have it taken up in the monitor system is to open a bug for it in the bugtracker, so please do so."

Well, i did... and this was the result..

[Proxy]

Oke (few weeks later) i now have another server installed.
CPU 3Ghz
1GB internal memory
etc..
Still using 12Mbyte (100Mb-it) local and internet eht0 - eth1

And now the conclusion:
I can now download and upload with -almost- 12Mbyte with proxy On!
So yes indeed it had to do with hardware!

SME Server is the Best! 

[Stefano]

...
CPU 3Ghz
1GB internal memory
...
So yes indeed it had to do with hardware!

well... good to know that I was right

Stefano