ISC Backup... The Kernel For A Dead-Images Remover??

Mod, Add-on & other Development discussion
Post Reply
CharlieFoxtrot
Confirmed
Confirmed
Posts: 413
Joined: Sun Aug 09, 2009 1:23 pm

ISC Backup... The Kernel For A Dead-Images Remover??

Post by CharlieFoxtrot »

PREFACE: My comments in this post are based on my experiences with ISC 4.07... things may have changed since this version (but I doubt it.)

It appears that when ISC makes a backup of the "product_images" directory, it copies ONLY the images that are associated with an product that's in the "isc_products" table. ~~ In other words: any "dead" or unused images in the "product_images" directory are not included in the backup.

After I discovered this... it occurred to me that the backup copy of "product_images" could be moved to the store's root directory (ie: as "product_images_new") and then quickly rename "product_images" to "product_images_bloated" ... then rename "product_images_new" to "product_images".

This works!! ~~ But with a few caveats! Take note!!!

1) The backup copy of "product_images" contains ONLY the active product images.

2) The code that creates the backup copy of "product_images" DOES NOT COPY THE FOLLOWING:

-- a) the category icons
-- b) the category default image (if you use something other than the default)
-- c) the store logo (if you've uploaded an image via the admin interface)
-- d) the "uploaded_images" directory.
-- e) the "wrap_images" directory.
-- f) the "configured_products" directory.

As a result of the MISSING ITEMS, my idea of replacing the bloated "product_images" with the backup "product_images" was a bit of failure.

The products displayed fine... but my store logo was missing and my category icons were missing.

It was simple enough to copy the missing directories into the new version of "product_images"... and I easily copied my store logo into the new directory as well. ~~ The biggest problem was the category icons!

Sometimes the ISC coding really impresses me... but other times, the short-sighted and LAZY programming just baffles me!! ~~ WHY ON EARTH would they not include the category icon images as part of the "product_images" backup?!!

They're IN the "product_images" directory already (scattered randomly through the "a-z" subdirectories) so they should also be included! Right?

MY SHORT TERM SOLUTION FOLLOWS...

For some people this may not be practical, but here's what I did.

1) I created a sub-directory in "product_images" called "caticons".
2) I peeked at the "categories" table and looked at the entries in the "catimagefile" field (this contains the filename and subdirectory name for every category icon.)
3) I cut-and-pasted the filenames into a text document for easy reading.
4) I copied all 150+ category icons into my new "caticons" sub-directory (one by one... painful)

Now... all my caticons are in ONE directory. ISC's backup script STILL won't include them during a backup... but at least I know they're ALL in the same place, and I can manually copy them when needed.

5) I made a backup copy of the "categories" table.
6) I edited the contents of the "catimagefile" fields and removed the "a-z" subdirectories... and changed it to "caticons".

For example:

a/computers_icon.gif
b/tires_icon.gif
c/mobile_icon.gif
...
z/printers_icon.jpg

became

caticons/computers_icon.gif
caticons/tires_icon.gif
caticons/mobile_icon.gif
...
caticons/printers_icon.jpg

THIS WORKED!!

NEXT... I knew that if I added any new category icons, they would ALSO be scattered randomly through the a-z subdirectories. So a quick modification to the PHP script was needed.

In the file: admin\includes\classes\class.category.php (around line 930) is the following code:

Code: Select all

// Determine the destination directory
$randomDir = strtolower(chr(rand(65, 90)));
I edited it to be

Code: Select all

// Determine the destination directory
// $randomDir = strtolower(chr(rand(65, 90)));
$randomDir = "caticon";
These small changes mean that every NEW category icon will be placed in the SAME DIRECTORY (caticons) instead of being scattered through the a-z directories... and then IGNORED by ISC's backup. ~~ By putting them all in the same directory, it will be much easier for me to copy/move them later on.

FOR NOW, I have two "product_images" directories in my store. ~ The old bloated one still exists as "product_images_bloated" until I'm convinced that my new version is working without any other problems.

In my next post... I'll talk about some ideas that are relatively easy for me to conceptualize... but that are far beyond my programming skills.
ISC 4.0.7

"... and let's be honest that whole "by design" thing is getting old too."
CharlieFoxtrot
Confirmed
Confirmed
Posts: 413
Joined: Sun Aug 09, 2009 1:23 pm

Re: ISC Backup... The Kernel For A Dead-Images Remover??

Post by CharlieFoxtrot »

The code for ISC's backup is flawed and incomplete... as I said before... it's shortsighted and lazy. However, the code that's there could be used as the kernel for something that could help with the image-bloat problem.

Using the round-about method described in my first post, I was able to reduce my image-bloat by 33%. (Well... the reduction will be realized when I'm confident enough to delete the "product_images_bloated" directory... but you get the idea.)

So... instead of coming up with code that will DELETE the unused product images, maybe it would be safer if the code would make a bloat-free copy of the "product_images" folder... WITH THE FOLLOWING CHANGES:

1) Instead of putting it in the "admin/backups" directory, it would be placed in the store's root directory.
2) It would be thorough enough to INCLUDE category icons and the other assorted files and directories that are currently overlooked.

Nothing would be deleted!! It would only make a copy. ~~ As the store administrator... we'd simply rename the two versions of the "product_images" folders and confirm that everything worked properly.

After we were convinced that the trimmed-down version was working properly (and assuming that we had not added any new products), then the old (bloated) directory could be deleted.
ISC 4.0.7

"... and let's be honest that whole "by design" thing is getting old too."
CharlieFoxtrot
Confirmed
Confirmed
Posts: 413
Joined: Sun Aug 09, 2009 1:23 pm

Re: ISC Backup... The Kernel For A Dead-Images Remover??

Post by CharlieFoxtrot »

Some final thoughts: Fortunately, my products are seldom deleted. Once listed, they remain for a long time. So it's taken me over two years to get the the point where I had 33% image-bloat.

I agree that my solution is TEDIOUS and CONVOLUTED!! (Taking a backup... renaming the backup... then copying the backup... then copying the missing files/folders into the backup, etc.)Totally insane!

But it's a chore that I'll only need to every couple of years (apparently). And now that I've edited the code so that all my category icons are isolated in a single subdirectory, it should be slightly faster when I do it again in 2013.

Well... that's it. Those are my observations and ideas. ~~ The changes I've described are beyond my hunt-and-peck (copy-and-paste) approach to PHP programming... but maybe someone will be able to do something with it.

I suspect that I'm not the only ISC user who has a problem with image bloat... I'm just glad that I can easily manage it.
ISC 4.0.7

"... and let's be honest that whole "by design" thing is getting old too."
Martin
Site Admin
Site Admin
Posts: 1854
Joined: Wed Jun 17, 2009 6:30 pm
Location: South Yorkshire UK
Contact:

Re: ISC Backup... The Kernel For A Dead-Images Remover??

Post by Martin »

Sometimes the ISC coding really impresses me... but other times, the short-sighted and LAZY programming just baffles me!! ~~ WHY ON EARTH would they not include the category icon images as part of the "product_images" backup?!!

They're IN the "product_images" directory already (scattered randomly through the "a-z" subdirectories) so they should also be included! Right?
Forget about why they've been lazy with the backup file... (although it does baffle me that it's such a half assed job)... I don't get the whole need for randomising the directory structure at all.. It's not like scrapers can't just spider the site as it is anyway and grab whatever they want regardless.

I did a quick look at the code and found that the use of this random directory system is used almost everywhere but not as part of a function so if you wanted to get rid of it, you'd need to search and replace every single call like this:

Code: Select all

$randomDir = strtolower(chr(rand(65, 90)));
Personally I'd be tempted to put a custom function like this in the library somewhere:

Code: Select all

// Destination directory
/*
 * Set destination directory either using a random choice (as per original ISC code)
 * or a choice based on the filename
 */
function setDestDir($filename=''){
	if(!empty($filename)) {
		// test first character of filename for a valid alphabetic character
		if(preg_match('/^[a-z]/', strtolower($filename), $matches) {
			return($matches[0]);
		}
	}

	/*
	 * If we can't find a valid character or we wanted a random character anyway
	 * return a randomised letter for the directory
	 */
	return(strtolower(chr(rand(65, 90))));
} 
... and then call it like this:

Code: Select all

$randomDir = ($_FILES['brandimagefile']['name']);
Post Reply