Title: "Google Merchant Static Feed"
Author: Martin Smallridge (WebSnail)
Cost: Paid
Support: 180 days
Purchasing: SnailSolutions
Version: 1.4.8 beta(revision 475-476)
Compatible: 6.0.11 - 6.1.8
ScreenCast: "Quick-n-Dirty" Backend Guide
Support Docs:
- Product listing with full description of what it does, doesn't do, etc...
- Avoid some common pitfalls with Google Merchant using this module
- How to setup the feeds on Google
This module exports your entire public product list into a Google Merchant (Froogle/GoogleBase) friendly format and writes it to a static file (instead of the dynamic one created in the admin version) so you can tell Google to grab this regularly.
If you set a CRON job (ie: scheduled task) to run the froogle.php file your file will be automatically re-generated according to the schedule you set.
Large Product Numbers:
This module has been re-engineered to provide a solution for those with ISC stores that contain
VERY large numbers of products (as well as smaller stores).
Previously the module (and Interspires own effort) would timeout or hit memory limit errors
causing the feed to fail without completing, making it all but useless.
The new functionality (version 1.1.3->) uses a number of approaches to make such large feeds
possible.
- Writes to a temporary feed file when generating a new feed.
(This leaves a completed feed intact and available for Google Merchant) - Detects when the temporary feed has not completed
- identifies the last successful product entry
- continues generating the feed from the next product
(It doesn't start from scratch if it doesn't need to)
- Only moves the new feed into the linked position when it has been completed
(Renames the older feed to [feedname].old so there's a fall back) - Handles the products in batches (user defineable in class.froogle.php) rather than getting
ALL the data from the database in one go. This reduces server memory load considerably - Uses a filelock system to ensure multiple instances of the module do not intefere with each
other, and/or corrupt the feed file being generated.
In short, it's seriously clever stuff (biased? me?

Security Issues?
Previously the froogle.php file could be used to cause a load issue on your server if someone chose to keep visiting it (or automated such a visit) to regenerate the feed and force your server to work constantly. To reduce mitigate any such "attacks" I've included a function that checks the feed age and refuses to regenerate the feed unless it's at least X ($file_min_age_secs) seconds old.
Unlike the previous version this feed generator will only list those products, attributes and bits of information already publicly viewable on your store. The feed will display (or not) these information fields dependent on whether you have chosen to show them as per your AdminCP > Display settings
- Quantity
- Brand
Product Code / SKU
If you wish to obscure the feed file location you can do so by editing the class.froogle.php file and changing the $exportFile variable to something a bit more obscure
eg: $exportFile = "DU32_ccvEds238DF.xml";
NB: Remember to included the .xml file extension or it will not be interpreted properly by google.
So, long as you remember to give Google the correct full path and filename you should be fine.
How to install
There is now a file called INSTALL.txt that has all the instructions for installation.
What to pass on to Froogle/GoogleBase
The URL for your product list will be
http:// yoursite.url/[store_directory?]/cache/[COUNTRYISO2]_froogle-export.xml
eg: yoursite.com/store/cache/GB_froogle-export.xml (for a UK feed)
If you've changed the value for $exportFile you will need to put that value in place of "froogle-export.xml"
eg: http:// yoursite.url/[store_directory?]/cache/[COUNTRYISO2]_DU32_ccvEds238DF.xml
Setup CRON
Ref: http://www.modwest.com/help/kb5-125.html
You need to set the path to the file properly so if you have a hosting account of "foo" with your store in the main/root directory you'd use this path:If your PHP scripts do not have executable permissions, 644 or -rw-r--r-- for example, then as part of the command portion of the cron line, you must specify the php interpreter and pass it the filename of the PHP script (including full path to the script from your home directory) that you want executed, like so:
The first cron line above will run myscript.phtml located in your home directory every day at 2:28PM.Code: Select all
28 14 * * * /usr/local/bin/php -q /home/account/public_html/store/froogle.php
/home/foo/public_html/froogle.php
Changelog
- Version 1.5.3 : Revision 546
- ADDED File: \modules\custom\class.progressbar.php
- UPDATED File: \froogle.php
- ADDED: Progress bar display using froogle.php?p=bar to activate.
Allows easy reference to see how much is left to generate - ADDED: Test for incomplete build(tmp) file. If over 2 weeks old it will now kill the file and force a new build.
This stops feeds wasting resources to build when a fresh start would be smarter move. - ADDED: Progress bar will correctly identify amount completed in an incomplete file and display progress accordingly
- FIXED: US taxonomy may be causing some issues when being read into the DB taxonomy file reordered so new index items are imported last.
- FIXED: Screen out categories where parent cats are not accessible/visible
- FIXED: Remove Undefined index error when no shipping option return from quote system
- Version 1.4.8 : Revision 475-476
- FIXED: Resolved empty feeds using 6.1.7 onwards
- ADDED: Free shipping and Fixed shipping product settings now interpreted correctly
- ADDED: More verbose information about products skipped and why
- FIXED: Resolved path check for Windows hosted ISC stores
- Version 1.4.7 : Revision 469-470
- UPDATED: Resolved incompatibility with 6.1.7 onwards
- UPDATED: 6.1.6 version of shoppingcomparison.categorybox.tpl included
- ADDED: Now pulls data from 6.1.7 google product search table if available
- UPDATED: US Google merchant taxonomy updated
- PENDING: Does NOT yet pull google product type from 6.1.7 so those versions onwards need to set the google product as a product comparison AND set something for google product search tab to ensure the product saves.
- Version 1.4.6 : Revision 428
- ADDED: Netherlands module for Product Comparison (Google Product Type selection)
- ADDED: Taxonomy for Netherlands
- Version 1.4.5 : Revision 403-404
- UPDATED: Tax information now retrieved and included for US States in US Feeds (when set)
- UPDATED: Tweaks to improve efficiency of tax data retrieval
- ADDED: Link to VidCap Screencast re: setting up GM Module
- FIX: Tmp feed files now auto deleted when unable to complete or id last product exported
(This will resolve problems where the presence of an error'd out TMP feed file blocks replacement)
- Version 1.4.4 : Revision 387
- ADDED: Option to read in Additional custom field information for 'color', 'gender', 'age_group', 'size'
- ADDED: field value limitations tested and only exported if valid for particular field label (eg: age_group = 'adult' or 'kids' only)
- FIX: Removed requirement to set postcode/state for any active feeds
- FIX: Add-on now shows the post/zip-code and state information when only one country stream/feed has been selected
- FIX: Custom field for MPN no longer has to have a lowercase field label (ie: 'mpn'). Can now be 'Mpn', 'MPN', etc..
- Version 1.4.3 : Revision 381-383
- ADDED: Ability to select state/county for each country feed (for shipping target)
- ADDED: Ability to input zip/post-code for each country feed (as above)
- FIX: Shipping quote now uses the new fields (above) resolving error with UPS and other shipping modules
- UPDATED: INSTALL.txt instructions to cover new shipping quote fields.
- Version 1.4.2 beta 1 : Revision 368-369
- Added Google Merchant Add-On which allows most common settings to be made in ISC Admin CP > AddOns > AddOn Settings
- Added option "Use Product Stock Codes (SKU) as MPN" (removes need to modify code)
- Added option "Include products that do not have an image"
- Added option "Override store Display setting for Brands" (removes need to modify code)
- No longer includes shipping costs if valid quote unavailable (Resolves missing "price" and "service" errors)
- Resolve PHP Notices regarding undefined variables
- Version 1.4.0 beta 2 : Revision 351
- Support for versions earlier than 6.0.11 not available (due to tax & google product version requirement)
- Availability attribute now resolved properly
- Now shows regardless of the quantity setting
- in stock when quantity shows in stock or if inventory tracking not enabled for product
- available for order if pre-order is on and date for availability is 2 days or less (2 days less than GM requires for safety!
- pre-order if date is later or not indicated
- out of stock for all other scenario
- UPC/GTIN/EAN improvements
- Trims the value of any leading/trailing spaces
- Checks for a comma separated value (eg: '123,789,12000') and uses the first value
- Rejects any values greater than the absolute max of 13 characters (ie: EAN, JAN, ISBN) and logs a warning.
- It will not validate the numeric for you...
- Shipping:
- Shipping quotes now provided for each product based on the cheapest shipping option available for that product to target country
- Tax
- Tax information now generated correctly for US feeds
- All other countries generate tax inclusive pricing (This assumes you have tax zones setup correctly!)
- Product Types
- Identified Bug in Category hierarchy generation system that stopped some products being given Product Types FIXED
- Google Product Type uses the productcomparison module code to set taxonomy data on categories with one module for each feed
- Google Product Type relies on settings at category level not individual products.
- Version 1.3.1 : Revision 331
- FIX: File lock and handle released when feed completed
- FIX: Feed Rename will now succeed (thanks to flock release)
- ADDED: Windows operating system/host now detected
- FIX: Windows feed path now backslashed to reduce issues with fopen()
- Version 1.3.0 : Revision 330
- FIX: Resolved issue for 5.5.0 - 5.5.4 so that product image links are now pulled out correctly
- ADDED: If catparentlist is NULL in categories table, code now generates the information from parentid data
- FIX: The above fixes an issue where <type> data wouldn't appear when catparentlist wasn't populated
- Version 1.2.6 RC#5 : Revision 316
- FIX: Resolves error "undefined index: 1"
- ADDED: Australia target feed option
- FIX: timeout issue with long feeds failing to publish/copy tmp feed to public feed location
- FIX: bad logic test caused by tmp feed "completing" but not published.
- FIX: "complete" tmp feeds will now publish on next run (as expected)
- ADDED: debug setting allows you to limit product numbers to be exported to 500 max.
(useful when trying to debug a VERY large product set) - ADDED: Logic test stops tmp feed corruption caused by browser refresh mid-feed generation
- Version 1.2.5 RC#4 : Revision 307
- FIX: Test for feed status failing due to file encoding incompatible with fgetc() php function
(Now resolved with new test)
. - FIX: Last product exported check failing for Incomplete feeds
- FIX: Last product exported check failing when dropiso2 used for single feeds
(Change in the ID field format now targetted with new Regex)
. - Default $file_min_age_secs (minimum feed age) setting now 29 minutes
. - Reformatted verbose output to provide clearer idea of success/failure
- New verbose error/success reporting added to feed status checks
(Makes troubleshooting much easier)
- FIX: Test for feed status failing due to file encoding incompatible with fgetc() php function
- Version 1.2.3 - 1.2.4 RC#3 : Revision 291
- DropISO2 option added for single feed users
(This drops the country code (eg GB) from the product id (eg: 25-GB -> 25 ) - brandExceptHide setting allows you to ALWAYS hide brand information for certain brands
- brandExceptShow setting allows you to ALWAYS show brand information for certain brands
(Setting uses the brandID, not the label, so you will need to look up ID in table to use) - FIX: Feed should complete properly now for very large product bases.
(Small issue with the logic caused the tmp feed to be reinitialised every time it was run) - (5.5.4 and earlier) Will get data from a custom field labelled "upc" (lowercase only!) and use as GTIN/UPC in feed (if data is available)
- DropISO2 option added for single feed users
- Version 1.2.2 RC#2 : Revision 281
- Foreign currency feeds now forced to use non-SEO links
(Trying to resolve an issue with google merchant returning 404 errors for setCurrency links) - Applied CNAME encoding for all field values
- Removed "model_number" as no longer used by Google Merchant
- MPN field/attribute now uses SKU/Product Code
(Code no longer 'makes up' an MPN as this is a huge GM policy infingement)
(Contact google.com/support/merchants/bin/request.py?contact_type=unique_id_exemption if you need an exemption) - Add $strictAtom to force application of strict Atom 1.0 spec' to feed
(ie: summary instead of description, link as <link href="foo"/> format not <link>foo</link> ) - Weight field/attribute now replaced by shipping_weight
- Expiration date now defaults to 29 days for local and 7 days for foreign currency feeds
(GM has a 3 day minimum for expiry)
- Foreign currency feeds now forced to use non-SEO links
- Version 1.2.1 : Revision 280
- [Removed extraneous $search introduced by code editor autocomplete]
- Minor adjustment to SQL ORDER BY for category retrieval
- Added sanity check for incorrectly formatted categoryparentlist field
(Stops extraneous blank product types appearing) - $exportFile setting/variable now in correct area to allow simple editing
- Fixes for ISC version 4:
- Doesn't export GTIN as not an available field
- $defaultCondition setting/variable allows a default "condition" to be specified
(Defaults to "new" if not set)
- $verbose default setting now set to false
- Version 1.2.0 : Revision 276
- Fix basix PHP syntax issue (array_key_exists())
- define ISC_CACHE_DIRECTORY if not defined. Resolves compatibility for earlier ISC versions
- Fix odd code insertion from code editor autocomplete (added $search variable)
- Resolve category/type fields retrieval properly
- Fix various bugs relating to product type/category and group specific visibility
- Version 1.1.9 : Revision 273
- Foreign Target feed links now correctly append setCurrency=x to link ( & or ? depending on whether query string already exists or not)
- Basic tax data included/excluded in price depending on tax zone (country/feed) ISC 6.x (Alpha Test Code!)
- Added invalid XML character strip function for "description" field
- 1.1.8 : Revision 268
- Added ISC database version detection function
- Fix: Product image links now provided in feed for products added since ISC version 6.x
- 1.1.7 : Revision 267
- Categories apply view permission based on default/guest group
- Properly hierarchy of visibility applied (parent/grandparent/etc.. category visibility affects child categories properly)
- Applied limit to number of product "type" fields can appear (limit 10) as per Google Merchant 10 max values per attrib limits
- 1.1.6 : Revision 264
- FIX: Force guest/default group permissions for product retrieval
(Avoids potential for feed to include group specific/hidden products appearing in feed during manual generation)
- FIX: Force guest/default group permissions for product retrieval
- 1.1.5 : Revision 262
- All user configurable variables are now commented in class.froogle.php so you know what they do
- FIX: Availability variable now indicates "in stock" if inventory tracking is not enabled for a product
- Feed files are tested for existence, read & write permissions
- Cache directory is tested for existence
- Sanity check against feed targets to ensure they are located in ISC Cache directory (to guard against hacker fishing expeditions)
- Feed write process attempts to set feeds to globally read/write-able
This should help with scenarios where files are not proving possible to rename/move when the feeds are owned by the apache or site/ftp account depending on whether generated by manual or CRON process.
Note: Deletion of the original feeds from cache directory may be required to make full use of this change.
- 1.1.4 : Revision 260
- Group specific products (hidden) included in feed
- Feed may not have completed properly due to total mismatch caused by above
- 1.1.3 : Revision 258
- Included $verbose variable to turn on/off echo'd debug output
- Updated lock functionality to exit gracefully where possible
- Added other country stream variables commented out
- Improved speed of "lastProductId" search
- Writes new feeds to .tmp file, leaving current feed intact until new feed is complete
- On completion current feed renamed to [foo].old while new feed renamed to primary location
- NOTE: This version SHOULD handle very large product lists (see above)
- 1.1.2 : Revision 256
- Added m_lock.php functions to allow stream file locking
- Added ability for code to recognise feed incomplete
- Code now identifies last successful product entry
- Code continues feed from next product and tries to complete
- Added additional echo messages to help with debugging
- 1.1.1 : Revision 87:
- Use fopen with 'a' flag to append data to end (efficiency)
- SELECT and fwrite in batches rather than the whole product database
- Module now checks for age of feed file and will only regenerate if older than $file_min_age_secs (seconds). This reduces likelihood of DDOS style attacks
- 1.1.0 : Revision 242:
- Massive overhaul of code
- TRUE multi-target feeds using unique ID (eg: 1-GB, 1-FR)
- Creates feeds for foreign currencies/markets only if the currency is defined in ISC currency settings
- Sale price (USA only) shows when applicable
- Departments deprecated/removed
- Product types added (based on category(s) associated with products)
- This gets rid of the old duplication issue we had before as well
- price/sale-price now includes currency code as a postfix
- brand depends on show config... (defaults to storename if not set)
- mpn depends on sku show config... (defaults to calc'd code from storesubstr, brandid, productid, etc...)
- gtin included if available (based on UPC field - ver 6 ISC)
- quantity depends on inventory show config
- shows availability if quantity not shown (in stock/ out of stock)
- No longer lists items that visitors do not have permission to see
- Limits title/product name to 70 characters (truncates to 67+... for anything longer)
- Sets expiry to 24 hours for foreign currency feeds to reduce chance of foreign exchange change discrepancies
- Tax attribute shown with zero value for US feeds for non-US stores
- 1.0.5 : Revision 87:
- GBP, EUR and USD specific lists now created
- exportFile remains as before for default currency
- skips any currencies that don't exist
- Prepends other currency files with the currency code eg: USD_froogle-export.xml
- No other changes necessary
- 1.0.4 : Revision 34:
- Update changes from ISC 4.0.9 that provide the "condition" field
- Default "condition" field to "new" if the condition field in database is not present or set (ie: backwards compatibility)
- Add $show_quantity option to show inventory quantities (set to true by default)
- Hide Quantity attribute for any items with a zero stock level even if $show_quantity is set to true
- 1.0.3 : Revision 31:
- Duplicate product list now optional using $ignore_duplicates variable in class.froogle.php
- Summary attribute/tag deprecated in favour of Description (as per documentation rather than out of date example on GoogleBase)
- Weight attribute/tag added for any products with weights set
- Tidied up formatting to produce a clean XML output
- More informative comments to help with editing, etc..
- 1.0.2 : Revision 29:
- Code now strips out duplicate product listings so that products only appear once instead of multiple times (if in multiple categories)
- Added revision, author and date information to make it easier to track changes
- 1.0.1 : Revision X:
- Fix invalid path issue when calling froogle from the CRON
- 1.0.0 : Revision X:
- Initial release