Jump to content
Sign in to follow this  
CMOTD

Easypopulate Data File Tools

Recommended Posts

Hm, dont get it, how do i have to work with your files? Do i have to upload them to my server... or do i have to install a php-installation local on my PC?

 

But sounds interesting, especially the tests of integrity of a file.

 

Greeting from Munich / Germany,

Sinarius :-)

Share this post


Link to post
Share on other sites

Yes, you need to install PHP locally.

 

Then you can run the scripts from the command line (make sure your PHP installation includes the command line option - can't remember if this is the default or if you have to specify it)

 

I noticed a post on the main easypopulate thread about date and time formats - I'll try and add a check for this soon - at the moment I'm working of getting rid of 'register globals'. nearly done !

 

regards,

 

Rich.

 

ps - Greeting from Basingstoke in the UK !

Share this post


Link to post
Share on other sites

I'm having real problems with the script.

 

common_config.php is this supposed to be renamed to common.config ? the readme reads like these config files are already contained in the zip.

 

Some of the files have <? php at the start and the space gives and error. some have <?php missing. are they supposed to ?

 

The only script i'm testing it with is import_data.php which is refusing to work even with the default data file and no modifications.

 

please help :blink:

 

g.

Share this post


Link to post
Share on other sites

ah i got the example data set to work adding an underscore to product_number

 

 

g.

Share this post


Link to post
Share on other sites

I've probably missed some vital functionality point but what are these binary files for ? edft edft2.txt...

 

g.

Share this post


Link to post
Share on other sites

Are you using the contribution I uploaded or are you using the contribution that David M. Graham "helpfully" uploaded using his toy operating system ?

 

Mr. Graham converted my tar file into a zip file and changed half the filenames in the process !

 

...hence his contribution won't work ! This is (as I am sure you can appreciate) very "helpful".

 

Assuming you ARE using Mr. Graham's contribution (it certainly sounds like you are), this might also explain why I haven't got a clue what you are talking about when you mention adding an underscore to product_number (where are you adding this ? - no don't tell me, unless you have the same problem in my contribution). I also have no idea what the edft and edft2.txt files are - they certainly have nothing to do with me !

 

I suggest you use the tar file I uploaded (v. 1.1 is the latest). This should work fine. Sorry, but I am not going to support someone else's hack.

 

Perhaps in the future, when Mr. Graham would like to be "helpful" again, he would do a decent job of it ? In the mantime, maybe he would like to offer support for his contribution.

 

regards,

 

Rich.

 

"...but like the Murphy's, I'm not bitter"

Edited by CMOTD

Share this post


Link to post
Share on other sites

sorry about all that i don't know why i downloaded that old version, it may have been from when i was collecting potential modules a while back.

 

I'm actually using your mod for a price syncronization mod where u can download supplier price lists and update the cost field in your db and use margin to calculate the price. It's working really well apart from the nasty datafiles i get from our suppliers.. 1 file has 2 lines of advertising at the start and the other doesn't have the field names at all. i can manually get this one to process by adding my own field line but the data itself has gaps with categories which your data importer uses as products.

 

Would i be best to process the file as part of the dataimporter or to use one of your post pre scripts, essentially all it needs is some check whether there are the right number of fields.

 

anyway tx again for a great mod.

 

g. B)

Share this post


Link to post
Share on other sites

a ha i see you've already added the ability to specify your own fields rather than take them from the file and because my script only updates prices which match a model number in the database any branding or categories that get read in won't effect the db :)

 

one thing i had to change was in :- import_data

 

$ep_record[$ep_x['EOREOR'][1]-1] = LINE_TERM;

to

$ep_record[$ep_x['EOREOR'][1]] = LINE_TERM;

 

because it wrote over the data in the second (of 2) fields of data that i use..

 

file_lib.php

 

//easy sync mod remove this check

if (file_exists(!$filename))

 

stopped files being written over.

 

anydo's it looks like it's going to be easy peasy thanx to your fine piece of work.

 

g.

Share this post


Link to post
Share on other sites

Thanks very much for the feedback.

 

I'm afraid I'm a bit unreliable at checking the boards so in future, if you (or anyone else out there) needs an answer from me within a sensible timespan then I suggest you email / pm me to give me a nudge to look at the board.

 

Do you think I ought to add an option to say how many title lines there are rather than just the option of 'no titles' or 'with titles' ? This would get round your problem with the two lines of advertising text I think.

 

I'll look at the points you've made - thanks again.

 

regards,

 

Rich.

Share this post


Link to post
Share on other sites

With regard to writing over existing files (re you having to remove ' if (file_exists(!$filename)'), there is a general poilicy within all these scripts to NOT overwrite existing files.

 

Maybe I need to add a -f (force) flag to override this poilicy - I can see that it could be annoying as it is.

 

Mmmm......

 

Rich.

Share this post


Link to post
Share on other sites

I've put up v 1.2 of this contribution.

 

This adds the correction to the EOREOR addition (many thanks to Giles Westwood for this)

 

Added some stuff to the image_resize script

 

I've not added the -f option that I mentioned previously - next time eh ?

 

regards,

 

Rich.

Share this post


Link to post
Share on other sites

Hi Rich,

 

First let me say thanks for that wonderful contribution :thumbsup:

 

I'm trying to use the split_data command on WinXP. After changing 2 constants in common.config

// Directory path deliminator of LOCAL machine (NOT your OSC server !). Set this
// to '/' for unix and '\\' for toy operating systems
define('LOCAL_DIR_DELIM', '\\');

// Regex version of above - '\/' for unix, '\\\\' for toy operating systems
define('LOCAL_DIR_DELIM_REGEX', '\\\\');

and then running the split_data command, I get the following output

Content-type: text/html

X-Powered-By: PHP/4.3.10

 

Reading data file 'EP2005Jan26-1010.txt'...

 

If any of the following info is wrong then there is probably something wrong

with the first line of the source data...

 

Info : I have determined that there should be 98 fields in the data.

Info : The file contains 31 records

Info : I have determined that there are 3 product name fields in the data.

Info : I have determined that there are 3 product description fields in the data.

Info : I have determined that there are 3 category fields in the data.

Line 2 : Wrong number of fields (22) - should be 98

Line 3 : Wrong number of fields (77) - should be 98

Line 23 : Wrong number of fields (72) - should be 98

Line 24 : Wrong number of fields (27) - should be 98

Line 26 : Wrong number of fields (10) - should be 98

Line 27 : Wrong number of fields (89) - should be 98

Line 31 : Wrong number of fields (52) - should be 98

Line 32 : Wrong number of fields (47) - should be 98

Outputting to splittest_000.txt... 31 records written

Done

I'm using an EP dump (complete tab-delimited .txt file) from the osCommerce standard installation.

 

If I import the EP dump in a spreadsheet the number of fields seem to be o.k. and also the EOREOR stuff is there at the end of every line.

 

Is there anything I might have overlooked or might there be a problem because I'm using the toy OS? :-"

 

P.S. The first line of my EP dump looks like this (tabs have been swallowed by the board):

products_model v_products_image v_products_name_1 v_products_description_1 v_products_url_1 v_products_name_2 v_products_description_2 v_products_url_2 v_products_name_3 v_products_description_3 v_products_url_3 v_products_price v_products_weight v_date_avail v_date_added v_products_quantity v_attribute_options_id_1 v_attribute_options_name_1_1 v_attribute_options_name_1_2 v_attribute_options_name_1_3 v_attribute_options_id_2 v_attribute_options_name_2_1 v_attribute_options_name_2_2 v_attribute_options_name_2_3 v_attribute_options_id_3 v_attribute_options_name_3_1 v_attribute_options_name_3_2 v_attribute_options_name_3_3 v_attribute_values_id_3_1 v_attribute_values_price_3_1 v_attribute_values_name_3_1_1 v_attribute_values_name_3_1_2 v_attribute_values_name_3_1_3 v_attribute_values_id_3_2 v_attribute_values_price_3_2 v_attribute_values_name_3_2_1 v_attribute_values_name_3_2_2 v_attribute_values_name_3_2_3 v_attribute_values_id_3_3 v_attribute_values_price_3_3 v_attribute_values_name_3_3_1 v_attribute_values_name_3_3_2 v_attribute_values_name_3_3_3 v_attribute_values_id_3_4 v_attribute_values_price_3_4 v_attribute_values_name_3_4_1 v_attribute_values_name_3_4_2 v_attribute_values_name_3_4_3 v_attribute_values_id_3_5 v_attribute_values_price_3_5 v_attribute_values_name_3_5_1 v_attribute_values_name_3_5_2 v_attribute_values_name_3_5_3 v_attribute_options_id_4 v_attribute_options_name_4_1 v_attribute_options_name_4_2 v_attribute_options_name_4_3 v_attribute_values_id_4_1 v_attribute_values_price_4_1 v_attribute_values_name_4_1_1 v_attribute_values_name_4_1_2 v_attribute_values_name_4_1_3 v_attribute_values_id_4_2 v_attribute_values_price_4_2 v_attribute_values_name_4_2_1 v_attribute_values_name_4_2_2 v_attribute_values_name_4_2_3 v_attribute_values_id_4_3 v_attribute_values_price_4_3 v_attribute_values_name_4_3_1 v_attribute_values_name_4_3_2 v_attribute_values_name_4_3_3 v_attribute_values_id_4_4 v_attribute_values_price_4_4 v_attribute_values_name_4_4_1 v_attribute_values_name_4_4_2 v_attribute_values_name_4_4_3 v_attribute_options_id_5 v_attribute_options_name_5_1 v_attribute_options_name_5_2 v_attribute_options_name_5_3 v_attribute_values_id_5_1 v_attribute_values_price_5_1 v_attribute_values_name_5_1_1 v_attribute_values_name_5_1_2 v_attribute_values_name_5_1_3 v_attribute_values_id_5_2 v_attribute_values_price_5_2 v_attribute_values_name_5_2_1 v_attribute_values_name_5_2_2 v_attribute_values_name_5_2_3 v_manufacturers_name v_categories_name_1 v_categories_name_2 v_categories_name_3 v_tax_class_title v_status EOREOR

What is the Matrix? The answer is out there, and it's looking for you, and it will find you if you want it to.

Share this post


Link to post
Share on other sites

Blimey ! You must be ...what ?.... the second (after me) person to use this !!

 

I think your configuration is probably ok.

 

The fact that you seem to be getting an almost random number of fields reported for the records would suggest that they are not formatted correctly (fields should be tab-deliminated).

 

The number of fields is reported as 98 and this would seem to be correct (I counted them in the last of the three snippets you posted).

 

Obvious thing to check : The record fields ARE seperated by tabs, yes ?

 

What I would do is load the csv file into a plain text editor. If the file is not too large, you should find notepad is actually ideal for this because (unlike some more sophisticated editors), it will preserve the tab characters and not try and convert them to spaces when it loads the file.

 

Then, pick a record that is being reported as faulty, and manually count the fields (ie - count the number of tab characters). Are there 98 ? I would guess that there are not.

 

Rich.

Share this post


Link to post
Share on other sites

CMOTD,

 

I am very much hoping that you can help me with a question that I cannot seem to get answered despite several posts in what I believe are the proper areas.

 

My question is largely related to Easy Populate and is very simple: Will Easy Populate continue to work properly even if I install the contribution entitled "Product Attributes - Options Type Feature" (shown here Link to Contrib)

 

I don't want to install this mod if Easy Populate will not recognize and download/upload the new attribute fields.

 

I have been very fortunate thus far in setting up my first store, despite my first exposure to a php-based site.

 

Now that it's almost entirely configured, I only have attributes and PayPal to finish up and I'm done.

 

I hate the thought of messing it up.

 

If you're able to help, it would be very much appreciated.

 

Thank you,

 

Lobo

Share this post


Link to post
Share on other sites
CMOTD,

...Will Easy Populate continue to work properly even if I install the contribution entitled "Product Attributes - Options Type Feature"

I have absolutely no idea.

 

...and this thread is not the place to ask - this is for issues with the EDFT contribution only.

 

Rich.

Share this post


Link to post
Share on other sites
What I would do is load the csv file into a plain text editor. If the file is not too large, you should find notepad is actually ideal for this because (unlike some more sophisticated editors), it will preserve the tab characters and not try and convert them to spaces when it loads the file.

 

Then, pick a record that is being reported as faulty, and manually count the fields (ie - count the number of tab characters). Are there 98 ? I would guess that there are not.

I put line 2 and also line 3 in a separate text file and then did a regexp search for \t

 

I always get 97 search results. So either I have a wrong configuration or there is a bug in your script when using WinXP?

 

I'm going to try this on my Red Hat box now...


What is the Matrix? The answer is out there, and it's looking for you, and it will find you if you want it to.

Share this post


Link to post
Share on other sites
I put line 2 and also line 3 in a separate text file and then did a regexp search for \t

 

I always get 97 search results. So either I have a wrong configuration or there is a bug in your script when using WinXP?

 

I'm going to try this on my Red Hat box now...

My mistake - 97 is correct, of course. And sorry for the notepad suggestion - you obviously know what you're doing. Makes a change :-)

 

Mmmm..... so what's cooking ?

 

I have never run these scripts on windows (and don't have the means to even if I wanted to), and as no one else seems to take any notice of this contribution, you are a pioneer as far as I know. Other than the directory deliminator change though, I can't see any reason why running it on windows would make any difference unless it's some obscure difference in PHP itself. But I think that's clutching at straws, eh ?

 

If you want to email me a field title line and a faulty record, I can have a look if you like (you'll have to stick them in an attached file of course to preserve the tabs).

 

Rich.

Share this post


Link to post
Share on other sites

Thought : It's not the end of line character(s) is it ? Another straw, but....

 

Can't see why this would be, but it's maybe worth a go.

 

You could try converting it to unix eol (there's a script in the contribution to do this if you don't have the means readily available)

 

Rich.

Share this post


Link to post
Share on other sites

Just for the record, the problems that Nick Weisser has been having have now been fixed. I'll upload a new version of EDFT when I get round to it. In the meantime, the problem is that the MAX_LINE_LEN value defined in base.config is (by default) set to 4000. This is not enough. Setting it to 10000 (or even 20000) is not unreasonable if you have large data sets.

 

Other problems that have been fixed :

 

If your scripts are timing out before they complete, try adding this to base.config :

php -d max_execution_time=0

...or maybe try this instead....

ini_set ("max_execution_time",999999);

One last thing. There is a typo in the split_data script help text. It says that the -d option is used in combination with the -c option. This is wrong - it is used in combination with the -t option. This is just a typo; the script should work fine.

 

Rich.

Edited by CMOTD

Share this post


Link to post
Share on other sites

Put up version 1.4 of EDFT. This addresses all known issues.

 

NOTE: I will no longer visit/monitor this forum. if you have problems then you can PM me or email me and I might get back to you. No promises though.

 

I will keep my PM / email address open but I don't know how long the forum will maintain my login (and hence my email address link) once it stops being used, so if you are reading this some time from now and find that my details have been erased then at least you know why.

 

bye.

 

Rich.

Share this post


Link to post
Share on other sites

fyi in case it helps anyone else...

 

I'm running my test system on a windows xp pro box and was having problems getting php to run the pre_edit script correctly.

 

C:\eclipse\oscommerce\edft_v1.4>php pre_edit EP2006Mar19-1010.txt
PHP Notice:  Undefined variable:  argc in C:\eclipse\oscommerce\edft_v1.4\pre_edit on line 65
PHP Notice:  Undefined variable:  argv in C:\eclipse\oscommerce\edft_v1.4\pre_edit on line 68
X-Powered-By: PHP/4.4.1
Content-type: text/html

Pre-edit processor. Takes an easypopulate data file (or block split files) and
performs a number of operations on it. These include:
...

 

No matter what I did, I got the argc/argv warnings and the default help message indicating I supplied no arguments.

 

Turns out my php.ini had argc/argv disabled. So, I used the php -n to not use php.ini and then it worked. :thumbsup: My example output is below. Granted this is from a small test site and the 4 products I exported using EasyPopulate have issues :>

 

C:\eclipse\oscommerce\edft_v1.4>php -n pre_edit EP2006Mar19-1010.txt
X-Powered-By: PHP/4.4.1
Content-type: text/html

Reading data file 'EP2006Mar19-1010.txt'...

If any of the following info is wrong then there is probably something wrong
with the first line of the source data...

Info : I have determined that there should be 34 fields in the data.
Info : The file contains 4 records
Info : I have determined that there is 1 product name field in the data.
Info : I have determined that there is 1 product description field in the data.
Info : I have determined that there are 7 category fields in the data.

Line 2 : Blank product number (v_products_model)
Line 3 : Blank product number (v_products_model)
Line 4 : Blank product number (v_products_model)
Line 5 : Blank product number (v_products_model)
Line 3 : Duplicate product image file (v_products_image) 'sotitle3.gif' - previous line = 2
Line 4 : Duplicate product image file (v_products_image) 'sotitle3.gif' - previous line = 3
Line 5 : Duplicate product image file (v_products_image) 'sotitle3.gif' - previous line = 4
Line 2 field 8 (v_date_avail) - blank date/time
Line 3 field 8 (v_date_avail) - blank date/time
Line 4 field 8 (v_date_avail) - blank date/time
Line 5 field 8 (v_date_avail) - blank date/time
Done

Edited by nfrobertson

Share this post


Link to post
Share on other sites

Could someone please tell me how to install php locally on WinXp pro and how to run those scripts? I'm only interested in using datafeed.txt to easy populate file conversion.

 

Thanks.

Share this post


Link to post
Share on other sites
Could someone please tell me how to install php locally on WinXp pro and how to run those scripts? I'm only interested in using datafeed.txt to easy populate file conversion.

 

Thanks.

 

The slickest local server for WinXP I've found is http://www.apachefriends.org/en/xampp-windows.html

 

It has an installer and is fairly simple to use then go here to get some answers.

http://www.apachefriends.org/en/faq-xampp-windows.html

 

HOWEVER, I have no idea how to run EPDF on it...but I'll let you know when I find out.


Sam M. - Seattle

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×