Jump to content

Archived

This topic is now archived and is closed to further replies.

Chemo

A Store Speed Optimization in Progress

Recommended Posts

- configuration flat file (cache)

- tep_get_tax_rate() is exited at beginning if my variable USE_OF_TAX = 0

 

Didier.

 

Where did you get these two contribs from Didier?

 

 

The JOIN is different for each module so can't give specifics unless you give me a specific file (i.e. - new_products.php)

 

Send me your modified file and I'll send one back with a few JOINS to get those queries down.

 

Whats your email address, would love it if you could do this for me!


---------------------------------

Duncan Jones

Share this post


Link to post
Share on other sites
Where did you get these two contribs from Didier?

Whats your email address, would love it if you could do this for me!

 

1. configuration flat file Faster Page Loads, Less DB queries

 

2. it's my own development. Very simple : change general.php

 

////

// Returns the tax rate for a zone / class

// TABLES: tax_rates, zones_to_geo_zones

function tep_get_tax_rate($class_id, $country_id = -1, $zone_id = -1) {

global $customer_zone_id, $customer_country_id;

 

// DDB - 040804 - DO not use taxes if not told so

if (USE_OF_TAX == 0) return 0;

if ( ($country_id == -1) && ($zone_id == -1) ) {

 

Just add a record in the configuration table for the parameter USE_OF_TAX (values 0-1).

 

Didier.

Share this post


Link to post
Share on other sites

You know, for those of you following along, there is a seriously flawed problem with the 'page cash' contribution. I've talked this through with 3 or 4 of the best programmers I know, and if you do not have 'force cookie use' set to true, there really isn't a good solution to fix it.

 

However, after doing quite a bit of testing, we've determined that it's not really the reduced unumber of queries that helps the load time of the page. It's actually the little bit of code that takes out all of the white space from the HTML output that makes it load so fast.

 

Before installing the page cache contribution, my site did 29 queries, and loaded in about 39 seconds on a 56k.

 

After installing the contribution, it had 9 queries, and loaded in 19.02 seconds.

 

After finding the flaw in the page cash contribution, I disabled the write to the cash file, and deleted all of the previous cache files. It now has 27 queries, and loadin in 22.17 seconds.

 

So, although the page cache really can't be fixed if you want to allow customers that don't have cookies enabled to be able to purchase, it's still pretty easy to use the html compression part of this contribution to make your page load faster.


-------------------------------------------------------------------------------------------------------------------------

NOTE: As of Oct 2006, I'm not as active in this forum as I used to be, but I still work with osC quite a bit.

If you have a question about any of my posts here, your best bet is to contact me though either Email or PM in my profile, and I'll be happy to help.

Share this post


Link to post
Share on other sites
You know, for those of you following along, there is a seriously flawed problem with the 'page cash' contribution. I've talked this through with 3 or 4 of the best programmers I know, and if you do not have 'force cookie use' set to true, there really isn't a good solution to fix it.

 

However, after doing quite a bit of testing, we've determined that it's not really the reduced unumber of queries that helps the load time of the page. It's actually the little bit of code that takes out all of the white space from the HTML output that makes it load so fast.

 

Before installing the page cache contribution, my site did 29 queries, and loaded in about 39 seconds on a 56k.

 

After installing the contribution, it had 9 queries, and loaded in 19.02 seconds.

 

After finding the flaw in the page cash contribution, I disabled the write to the cash file, and deleted all of the previous cache files. It now has 27 queries, and loadin in 22.17 seconds.

 

So, although the page cache really can't be fixed if you want to allow customers that don't have cookies enabled to be able to purchase, it's still pretty easy to use the html compression part of this contribution to make your page load faster.

I added the compression (as you noticed it is actually just removing the whitespace) and haven't even mentioned it. It's nice to see someone that still looks at the code to see what's under the hood :)

 

I think there may be a simple solution to the session id problem. It is the same thing I faced when trying to create a cached page everyone can use yet still keep a portion dynamic (the shopping cart). The session id is set in application_top.php so why not do the same thing as with the shopping cart info?

 

On writing the cache file remove the session id and insert placeholder. On cache page retrieval str_replace() the placeholder with the actual session id. I'll get on it and see what I can come up with.

Share this post


Link to post
Share on other sites

I just uploaded v1.2 of the page cache contribution. The session_id support is in there now :)

 

Here's how I did it:

 

Class constructor

global $SID
$this->customer_sid = $SID;  

 

cache_this_page() method

...
echo str_replace(array("<%CART_CACHE%>", "<osCsid>"), array($this->customer_cart, $this->customer_sid), file_get_contents($this->cache_file) );    
...

 

write_file() method

$buffer = preg_replace("/osCsid=.{32}/", "<osCsid>", $buffer);

 

So, on new load it outputs the osCsid like normal. On cache file write it removes the osCid from all the URL's and replaces it with a placeholder. On subsequent calls to the cache page it inserts the $SID in place.

 

Now, how about having some people test it? :)

Share this post


Link to post
Share on other sites
Before installing the page cache contribution, my site did 29 queries, and loaded in about 39 seconds on a 56k.

 

After installing the contribution, it had 9 queries, and loaded in 19.02 seconds.

 

After finding the flaw in the page cash contribution, I disabled the write to the cash file, and deleted all of the previous cache files. It now has 27 queries, and loadin in 22.17 seconds

 

I hear what you are saying but don't think that your own load time tests on a 56k modem and on one solitary site can be a sound basis for testing.

 

With so many different isp's, backbones and servers coupled with so many different client side computers - surely the important thing for all of us is the Parse Time. With this we then KNOW how fast or slow OScommerce is, irrespective of different internet connections and your own image sizes etc etc.

Share this post


Link to post
Share on other sites
So, on new load it outputs the osCsid like normal. On cache file write it removes the osCid from all the URL's and replaces it with a placeholder. On subsequent calls to the cache page it inserts the $SID in place.

 

Now, how about having some people test it? 

 

That sounds like a pretty good solution. I wish I had thoguht of it.

 

I'll give it a whirl.

 

With so many different isp's, backbones and servers coupled with so many different client side computers - surely the important thing for all of us is the Parse Time. With this we then KNOW how fast or slow OScommerce is, irrespective of different internet connections and your own image sizes etc etc.

 

I guess that's true, but I saw less that on tenth of a second difference from before the contrib to after it. I think that testing load time on your own sever, being consistant on the tool you use is a much better guage of the improvements each owner is making.


-------------------------------------------------------------------------------------------------------------------------

NOTE: As of Oct 2006, I'm not as active in this forum as I used to be, but I still work with osC quite a bit.

If you have a question about any of my posts here, your best bet is to contact me though either Email or PM in my profile, and I'll be happy to help.

Share this post


Link to post
Share on other sites

I just installed this contribution and got a couple of questions:

 

This might be a stupid question, but will this affect my statistic?

I am using advlogger and have the cgi script called from each page through a line like this: a.src="/advlogger/logger.cgi?ref="+escape(refer)+"&doc="+escape(doc)+"&res="+x+"x"+y;

Will it still call the script from a cached page?

 

 

What lifetime is best to use?

It's preset to 5 minutes, but why not 1 or 2 hours hours or maybe more?

 

 

Regards

Mike

Share this post


Link to post
Share on other sites

One more thing....

The shopping cart in left column doesn't show up before the customer is logged in.

When the customer is not logged in and order a product with "buy now' he can't see if it is actually added to the cart, unless he actually go to the cart and check it out.

Any way around that? Maybe some kind of pop-up confirmation when a item is placed in the cart....

Share this post


Link to post
Share on other sites
Here are the options:

1. Disable default appending session_id's to URL's setting in tep_href_link()

2. Create code to strip the session id's from the cache code

 

Here is what my tep_href_link() looks like:

 ?function tep_href_link($page = '', $parameters = '', $connection = 'NONSSL', $add_session_id = false, $search_engine_safe = true) {

Now, even if the session_id's are not appended to the URL they will still be created for guests uniquely.  Once they log in they get another or resume a previous.  Once they put something in their cart the session_id is recorded in session and that is good for the life of their visit.  If they log in they will get a more permanent session_id.

 

For some Reason when I make this change...

function tep_href_link($page = '', $parameters = '', $connection = 'NONSSL', $add_session_id = false, $search_engine_safe = true) {

 

I can not log in anymore. What do you think could be causing this?


:)

Share this post


Link to post
Share on other sites

I have been working on the products_name and count(*) queries :

 

1. this is a query of my product_info.php page to read all options. Ofter another query is done to count the number of records before (like select count(*) as total from...). You can put all in 1 query and use it this way :

 

$query = teb_db_query(...);

$count = tep_db_num_rows($query);

if ($count >0)

{

$query_row = tep_db_fetch_array($query)

...

}

 

 

2. this is another query for modules likes also_purchased_products where with 1 query you can drop the additional calls to function tep_get_products_name().

 

$orders_query = tep_db_query("select p.products_id, p.products_tax_class_id, p.products_price, pd.products_name from " . TABLE_ORDERS_PRODUCTS . " opa, " . TABLE_ORDERS_PRODUCTS . " opb, " . TABLE_ORDERS . " o, " . TABLE_PRODUCTS_DESCRIPTION . " pd, " . TABLE_PRODUCTS . " p where opa.products_id = '" . (int)$HTTP_GET_VARS['products_id'] . "' and opa.orders_id = opb.orders_id and opb.products_id != '" . (int)$HTTP_GET_VARS['products_id'] . "' and opb.products_id = p.products_id and opb.orders_id = o.orders_id and p.products_status = '1' and pd.products_id = p.products_id and pd.language_id = '". (int)$languages_id . "' group by p.products_id order by o.date_purchased desc limit " . MAX_DISPLAY_ALSO_PURCHASED);

 

Now I just have to get rid off the queries for the "specials". A lot of work but the query count is getting down !!

 

Didier.

Share this post


Link to post
Share on other sites

So far I'm at:

 

Home page: Page Parse Time: 0.054s with 9 queries

Category showing products: Page Parse Time: 0.048s with 10 queries

Product page: Page Parse Time: 0.050s with 13 queries

 

Cool huh? :) Thanks Chemo!

 

I have your latest "Page Cache 1.2" installed. Still reading this post for more ideas.

Share this post


Link to post
Share on other sites

A Random Product:

 

First Time Load - Current Parse Time: 0.180 s with 73 queries

Reload - Current Parse Time: 0.044 s with 12 queries

 

====

 

Thanks Alot!!

 

One thing, do you think having the parse time showing and the code in the page would make any difference to load time :P


---------------------------------

Duncan Jones

Share this post


Link to post
Share on other sites
I can not log in anymore. What do you think could be causing this?

 

Don't make this change. It's not needed anymore because he has fixed the SID issue in the latest release ver. 1.2. Get the new version of Page Cache.

 

Has anyone tested the new version to make sure it's no longer a security problem? I would, but have no clue what to look for. :)

Share this post


Link to post
Share on other sites

I hate to rain on everyone's parade but there is still work to be done with the page cache contribution.

 

The session id needs to be passed to every page request since it is still being cached in some instances. The solve all answer is to pass a place holder that will be cached and insert session_id on each page request. If cookies are enabled on the customer's browser it does not append the session_id to the URL. In other words, function exactly like the stock osC installation. Currently, if cookies are enabled on the customer's browser it does not append the session_id but still has the ? after each URL. We need to clean that up.

 

Further development needs to be done before the contribution should be used on a production store. Alas, this is the burden that is shared with any worthwhile contribution.

 

Right now I am working on 2 paid projects so cannot afford volunteer time to the project. Are there any devs that can look under the hood and help the cause? The base class is simple and is not hard to follow...plus it THOROUGHLY commented.

 

So, what needs to be done to make this production quality is to clean up the session_id replacement. Anyone want to give it a shot?

 

PROBLEM:

On page cache file write it replaces the session_id with a placeholder. If no session_id is passed the placeholder is not passed either. This is presented when a cookied customer calls a non-cached page and the cache is written without the placeholder. On subsequent calls to the cache page from browsers WITHOUT cookies enabled it does not append the session_id properly.

 

SOLUTION:

Pass the placeholder on each call and replace with session_id. If the customer is cookie enabled then do not append the session_id. If cookies are NOT enabled then append normally.

Share this post


Link to post
Share on other sites
One thing, do you think having the parse time showing and the code in the page would make any difference to load time :P

 

LOL, I thought about this also. The code to show you this must slow down the page by a couple milli-seconds or so. So let's get a contribution to speed this up a little. ;)

Share this post


Link to post
Share on other sites

Booked for the weekend now as well.

 

Perhaps next week?


-------------------------------------------------------------------------------------------------------------------------

NOTE: As of Oct 2006, I'm not as active in this forum as I used to be, but I still work with osC quite a bit.

If you have a question about any of my posts here, your best bet is to contact me though either Email or PM in my profile, and I'll be happy to help.

Share this post


Link to post
Share on other sites
Don't make this change.  It's not needed anymore because he has fixed the SID issue in the latest release ver. 1.2.  Get the new version of Page Cache.

 

Has anyone tested the new version to make sure it's no longer a security problem?  I would, but have no clue what to look for. :)

The session_id fix in the last version is still flawed...more work needs to be done.

 

This project needs help if we're going to get this to a production level. We need developers to take a look and fix this last issue before it'll be completely bug free. As I said above, I'm on 2 paid projects that have to be done first so I can't get on it for another week or so (I've been away from the board and not replying to the thread because of this too).

 

So, one last bug to fix and we'll all have faster stores :)

Share this post


Link to post
Share on other sites

I'm interested, but like I said, I booked until at least Tuesday.


-------------------------------------------------------------------------------------------------------------------------

NOTE: As of Oct 2006, I'm not as active in this forum as I used to be, but I still work with osC quite a bit.

If you have a question about any of my posts here, your best bet is to contact me though either Email or PM in my profile, and I'll be happy to help.

Share this post


Link to post
Share on other sites

Well, I have a few minutes since the family already went to bed so I'm over on the dev site now. I have it so the session_id's are being passed correctly to both cookied and NON-cookied browsers but still needs a bit of work.

 

As it is now on the dev site all pages are cached with the sid placeholder. Now, when a cookied browser hits the cache file it still outputs the separator (but no sid appended as it should be). I'll spend a few minutes cleaning it up and may be able to release something tonight or early next week.

Share this post


Link to post
Share on other sites
Well, I have a few minutes since the family already went to bed so I'm over on the dev site now.  I have it so the session_id's are being passed correctly to both cookied and NON-cookied browsers but still needs a bit of work.

 

As it is now on the dev site all pages are cached with the sid placeholder.  Now, when a cookied browser hits the cache file it still outputs the separator (but no sid appended as it should be).  I'll spend a few minutes cleaning it up and may be able to release something tonight or early next week.

 

 

Sounds Great Chemo I for 1 appreciate all your hard work and dedication. :thumbsup:


:)

Share this post


Link to post
Share on other sites

The suggestions that I have found in this thread have cut down ALOT on the load time. I have not used the cache thing as it kind of worries me till the bugs are worked out. Also, I do not know how to do the join commands as I am new to php. Would it be possible for you to upload a copy of the modified file(s) with the join commands and add comments in the file as to where the changes for that start and stop, then I could use Beyond Compare to compare them with my files and make the changes?

Share this post


Link to post
Share on other sites

Alright...I have the sid issue resolved.

 

I'll upload a new version tomorrow or Sunday.

 

This fix involves 4 file changes (nothing major) and affect the following files:

 

1. includes/application_top.php

2. includes/header.php

3. includes/functions/html_output.php

3. includes/classes/page_cache.php

 

Basically, the fix involves revising this part of tep_href_link() to read as follows:

 ? ?if (!tep_session_is_registered('customer_id') && ENABLE_PAGE_CACHE == 'true' && class_exists('page_cache')) {
?$link .= $separator . '<osCsid>';
} elseif (isset($_sid)) {
? ? ?$link .= $separator . $_sid;
? ?}

It checks to see if the customer is logged in, the page cache is enabled, and finally whether the class exists. If so it returns the placeholder for the sid.

 

After this change the page_cache.php class was changed to replace the placeholder with the $SID.

 

The next bug that came up was certain parts of the page are constructed BEFORE the page is cached (affecting application_top.php). The elements are the breadcrumb trail which is set near the bottom of the file. Not a big problem...just move that part of the code to the top of header.php. That way it'll still work the same on all pages and still be cached.

 

That's it...

 

Now, I need to get some people over on the dev site and test this out before release of this version. Here's how to test:

 

First, set the browser to not accept cookies from the domain. Then proceed through the checkout process. Look for any errors like malformed URL's or items dropping from the cart. use the mouse to hover over links to check their form. If everything goes well do the same with a cookie enabled browser.

 

If both check out then report back with an "A" OK and I'll release the version...

 

Come on people...if you want this revised code you have to help me test it first! :)

 

 

Happy bug hunting!

Share this post


Link to post
Share on other sites
The suggestions that I have found in this thread have cut down ALOT on the load time.  I have not used the cache thing as it kind of worries me till the bugs are worked out.  Also, I do not know how to do the join commands as I am new to php.  Would it be possible for you to upload a copy of the modified file(s) with the join commands and add comments in the file as to where the changes for that start and stop, then I could use Beyond Compare to compare them with my files and make the changes?

Sure...as soon as the page cache is tested and released I'll write a nice tutorial on JOIN commands and where they could potentially be used with osCommerce. I think a general "how-to" will serve the community better than a specific example. That way the knowledge could be applied to several scenarios and used over and over.

 

Put that knowledge together with the debug / query output contribution and you're well on your way to trimming the fat.

Share this post


Link to post
Share on other sites

Ok, I just ran through two orders on the test site. I am using netscape 7.2. The first time I went through after having set the browser to deny all cookes from the site and had no issues, then I went through with cookies allowed and still had no problems.

Share this post


Link to post
Share on other sites

×