To JPEG or not to JPEG, The Good, The Bad and the Ugly

Running the same basic tests after moving the A7r MkII RAWs to an NVMe partition made the above processes so fast that I could not even estimate timing. Of course, ACR would also have already cached them, which would partly explain the almost instantaneous loads. Maybe I should try this after purging the ACR cache?
 
Ovi, I've just downloaded 6 RAW files from DPR's tests of this camera (Sony A7r MkII). These are located on my new Seagate 4 TB Ironwolf HDD, NOT on my NVMe SSD. These are 41 MB each, approx.

In Bridge, it takes about 1-2 seconds to open the preview at 100%. From there, ACR opens the image all but instantaneously in ProPhotoRGB 16 bit.

Click to open in Photoshop CC 2023, and again, it's all but instantaneous.

Both the Bridge and ACR caches are on the 1 TB NVMe SSD, and it's extremely fast. Ditto for the Photoshop scratch disk.

Now, I'm running all sorts of other crap as well as these programs, but physical memory use is just over 17 GB while doing the above. Stopping Outlook, Brave browser, PIE Studio, FastStone Viewer and OM Workspace and a few other miscellaneous programs would have brought this well under 16 GB of physical memory.

I do not know how Lightroom handles its database, this might be your problem?

Have you tried using Bridge and Photoshop CC? If so, what were the results?
I have not tried, Photoshop is not something I am really comfortable with (especially compared to Lightroom), I only know the more advanced features that I can't get in Lightroom like Content-Aware Fill, Focus Stacking, Cloning and Healing. I don't even use the Layers, to be honest.

I imported the wedding files last night and edited a few images (before going to bed):
Log.CSV (it's a CVS file)

The start page on the Physical Memory Used is the default usage when starting Windows up with all my "other things" running, it's at 10GB RAM.
The drop to 7.6 GB RAM usage is from closing all of those (Including Microsoft Edge, the web browser service runs in the background even if you close it, it needs END TASK in Task Manager to shut it down).
The usage going back up to 8.8 GB is from opening Lightroom with a new (empty) catalogue.
The jump to 14-15GB is from loading up the import window and going back down to 13GB is from importing (Copy RAW file) and then Building 1:1 Previews.
The jump back to 15GB of RAM at 59:11:2 on the timeframe of the log is when I started editing the pictures, with 1 to 2 GB of spare Physical RAM memory to address "if needed".
The drop to under 10GB RAM usage at 23:34:5 in the log is when I started using Topaz DeNoise AI, which uses a .TIFF file, made by Lightroom.

The performance was mildly frustrating (not to the breaking point but that's because I edited only 6 images back to back) but I started getting the lagging of controls, waiting for Lightroom to respond/preview the changes, and a few black screen lockups that usually take a minute or two to "unlock" itself. On the last one it's usually when Windows goes through the Page File System (and even though I have a really fast nVME, I think it's 3GB read and 1.7GB write per second) it's when my mood really falls off the cliff and my frustration with editing the picture puts me really off.
All other temperature and usage starts are normal I have come to expect for the last 4 years with the laptop.
 
OK. That really does sound memory bound.

However, my NVMe is running at 6-7,000 MBs for both read and write, so well over double the speed.

Can you replace yours with something newer/faster? Or add a second newer/faster one and clone your existing NVMe SSD onto it, then use your existing drive for parking image files, etc?
 
OK. That really does sound memory bound.

However, my NVMe is running at 6-7,000 MBs for both read and write, so well over double the speed.

Can you replace yours with something newer/faster? Or add a second newer/faster one and clone your existing NVMe SSD onto it, then use your existing drive for parking image files, etc?
1667041409950.png

My platform is limited to PCI Express 3.0, my Corsair Force MP510 2TB is about the limit of what I can get:
1667041520051.png

Even if I had PCI Express 4.0 the cost of a 2TB nVME at those top speeds would be about as expensive as upgrading my motherboard with i9 and 32GB of RAM. Unforunetly I do not have 2 m.2 slots, which is another travesty from ASUS on a device of this size and price (even for 2018 release date).
So right now I have a few options but they are all imbalanced on the cost of implenting them (and I don't mean just £):
*Lower MP camera (Sony a7C, a7 III, a7 IV, a9)
*Convert the RAW files to a more memory friendly file, maybe .TIFF or a downsized .DNG (I haven't explored this idea just yet)
*Shoot JPEG instead
*Upgrade the motherboard to i9 9980HK and 32GB DDR4
*A new/used seperate editing device

(I do wish I could opt in the camera for smaller RAW file format but it took Sony ONLY 9 years and 15 different a7/9/1 camera releases to reach to the a7R V to actually offer this !!!)
 
Also, switch off BUILD 1:1 PREVIEWS.

This is a killer for Bridge and I imagine for Lightroom.

Set your maximum and minimum virtual memory file to the same size. I suggest 16/16 GB as a starting point. Reboot the system after doing this. NEVER allow Windows to manage its own swap file!

Memory limits really kill Adobe products! You need to review how much memory they can use in SETTINGS in the program.

You may be well served getting one of those I9 motherboards with 32 GB RAM. Can you fit it yourself? What speed does it run the bus at? How many channels?

Other memory hogs are Edge and Windows Explorer ...

The other alternative may be to sell your laptop and build a relatively high performance PC, as I have done. Might be cheaper, and better.

I just checked my MoBo manual, and it appears that it has PCIe 5.x and PCIe 4.x slots, running 4 channels.

I'm going to sleep now. Talk further tomorrow.
 
Also, switch off BUILD 1:1 PREVIEWS.
1:1 Previews were supposed to help speed up the process for very large RAW files because it gives you a smaller (and a bit lesser quality) preview to speed up the response time and the editing process.
This is a killer for Bridge and I imagine for Lightroom.

Set your maximum and minimum virtual memory file to the same size. I suggest 16/16 GB as a starting point. Reboot the system after doing this. NEVER allow Windows to manage its own swap file!
For the Page File System, I have fixed a minimum and maximum of 100GB because sometimes I work with very large files that only hundreds of GB or RAM could, realistically, handle them. I have tried a smaller Page File but it hasn't changed the performance at all.
Memory limits really kill Adobe products! You need to review how much memory they can use in SETTINGS in the program.
I have it set at 90% of 16GB, which is 15.5 GB. That's why I frequently hit just below that metric when I edit pictures.
You may be well served getting one of those I9 motherboards with 32 GB RAM. Can you fit it yourself? What speed does it run the bus at? How many channels?
It's a consumer i9 chip so it's limited to 2 channels for memory (you need HEDT pr Threadripper for quad or even 6 channels for memory). Yes, the one good thing about this laptop is that's very easy to disassemble down to the motherboard. I have disassembled it just a few weeks ago to clean and repaste everything.
Other memory hogs are Edge and Windows Explorer ...
I do End Task for Microsoft Edge and I stopped watching YouTube videos on the laptop while editing. I am a bit more apprehensive to shut down Windows Explorer (if that's the one you meant instead of Internet Explorer, which has been deprecated and removed on Windows 11) because it might cause instability in the UI of Windows and affect its ability to read/write files during the editing process.
The other alternative may be to sell your laptop and build a relatively high performance PC, as I have done. Might be cheaper, and better.
I thought about it but Windows laptops have lesser selling value than Apple machines unfortunately and I don't think I would get more than 1.000 £ or even less if I tried to sell it (privately or not). And I would feel remorse for losing the ability to use my active pen for making local edits (the 49% reason why I bought this laptop machine), especially for portraits. To make a similar system on the Apple side I would need an M1 device, either Mac Mini or a MacBook, and an iPad with Apple Pencil to work in tandem via Apple's Sidecar feature. (I am unable to use graphic tablets because I can't get my brain to hand coordinate when I can't see the hand/pen on the same focal plane.)
I just checked my MoBo manual, and it appears that it has PCIe 5.x and PCIe 4.x slots, running 4 channels.
Yes, the "new" 12th Gen from Intel and 5th Gen from AMD support PCI Express 5 and on nVME storage you can get in excess of 10GB per second storage which is reaching the limits of what the OS and software can do with those speeds, but it's still nowhere near the speeds of RAM and the effectiveness of more RAM on heavy memory workloads. I did the mistake of believing people that 16GB of RAM is more than enough when I knew myself that 32GB will be the minimum for my own needs and workloads within the 5 years life span I had for the laptop. Now I am finding myself (and expecting) that 20GB RAM is the base I would need and it will grow to 32-40GB in the next 5 years. So anyone who wants to make a future-resistant (there's no such thing as future-proofing) editing device would be 32GB at the minimum and 64GB recommended.
I'm going to sleep now. Talk further tomorrow.
Thank you so much for the advice, I really appreciate it, see you tomorrow.
 
PS. This is the level of which I have "optimised" (aka Killed Processes) RAM usage before I start editing pictures:
1667045131836.png

Most of the Running programms (by Memory Allocation) is Window's own processes, nVidia GPU drivers, ASUS propriatary drivers for this laptop and Google's Drive and Microsoft's One Drive.
 
Update on editing the wedding pictures. The laptop behaved most of the time, there were a few hiccups when cropping or trying to clone stuff within Lightroom, had a few system freeze-ups for a few minutes. But I still feel like editing 90 pictures out of hundreds shouldn't have taken a whole day. The RAM usage was a consistent 13 to 15 GB (depending if I was working with .AWR file or .TIFF from external processing).
 
1:1 Previews were supposed to help speed up the process for very large RAW files because it gives you a smaller (and a bit lesser quality) preview to speed up the response time and the editing process.
My experience is that it does the exact opposite. Retrieval from NVMe SSDs and SATA drives with big caches is much faster than any time saved by doing this during ingestion. It just slows down ingestion (horribly so).
For the Page File System, I have fixed a minimum and maximum of 100GB because sometimes I work with very large files that only hundreds of GB or RAM could, realistically, handle them. I have tried a smaller Page File but it hasn't changed the performance at all.
I would cut this back to 16/16. Windows will automatically expand the VM file as needed, but fragments it all over the place! When you reboot, it goes back to the set size, which should be one contiguous file. If it isn't, do a boot time defrag to defragment all the system files. The effect on SSD life is minuscule, unless you do it every few minutes ...
I have it set at 90% of 16GB, which is 15.5 GB. That's why I frequently hit just below that metric when I edit pictures.

It's a consumer i9 chip so it's limited to 2 channels for memory (you need HEDT pr Threadripper for quad or even 6 channels for memory). Yes, the one good thing about this laptop is that's very easy to disassemble down to the motherboard. I have disassembled it just a few weeks ago to clean and repaste everything.
Probably not worth the cost of the changeover then. Even though the 32 GB would almost certainly alleviate your memory problem. Better off putting that money towards either a desktop, or selling your laptop and buying a more modern one. Seems like you had the misfortune to buy right at the end of a technology cycle, rather than at the beginning of one, which I have hopefully done - more by chance than planning!
I do End Task for Microsoft Edge and I stopped watching YouTube videos on the laptop while editing. I am a bit more apprehensive to shut down Windows Explorer (if that's the one you meant instead of Internet Explorer, which has been deprecated and removed on Windows 11) because it might cause instability in the UI of Windows and affect its ability to read/write files during the editing process.
Do not stop the Windows Explorer process in Task Manager. If you do, you can run it again from TM. It's a system process. But do shut down all unnecessary invocations of Windows Explorer from the GUI. They can be real memory hogs.
I thought about it but Windows laptops have lesser selling value than Apple machines unfortunately and I don't think I would get more than 1.000 £ or even less if I tried to sell it (privately or not). And I would feel remorse for losing the ability to use my active pen for making local edits (the 49% reason why I bought this laptop machine), especially for portraits. To make a similar system on the Apple side I would need an M1 device, either Mac Mini or a MacBook, and an iPad with Apple Pencil to work in tandem via Apple's Sidecar feature. (I am unable to use graphic tablets because I can't get my brain to hand coordinate when I can't see the hand/pen on the same focal plane.)

Yes, the "new" 12th Gen from Intel and 5th Gen from AMD support PCI Express 5 and on nVME storage you can get in excess of 10GB per second storage which is reaching the limits of what the OS and software can do with those speeds, but it's still nowhere near the speeds of RAM and the effectiveness of more RAM on heavy memory workloads. I did the mistake of believing people that 16GB of RAM is more than enough when I knew myself that 32GB will be the minimum for my own needs and workloads within the 5 years life span I had for the laptop.
Yes. The dangers of the internet. Everyone's an 'expert'. Most patently aren't, but occasionally some received wisdom gets through that's just unicorn droppings. The "16 GB RAM is enough" story is one of the latter. It isn't.

With the right hardware throughout the computer, 32 GB is fine. My new PC is constantly nudging 16-19 GB used. If (when ... ) I upgrade the RAM, it will be 2x 32 GB chips.
Now I am finding myself (and expecting) that 20GB RAM is the base I would need and it will grow to 32-40GB in the next 5 years. So anyone who wants to make a future-resistant (there's no such thing as future-proofing) editing device would be 32GB at the minimum and 64GB recommended.
Totally agree. Particularly if you are stuck with your initial choice, I would recommend 64 GB.
Thank you so much for the advice, I really appreciate it, see you tomorrow.
Only too happy to be of some little help, Ovi. Even if it's only as an intelligent sounding board to help you get your own thoughts in order.

I've spent nearly two years thinking about my own upgrade, which has been desperately needed for all that time. Other things prevented it from happening. When I finally plonked down the cash (nearly AU$3,000 with a couple of bits of software, but no OS, HDDs, keyboard, mouse or monitor) there was still quite a lot of trepidation at the back of my mind. With a small amount of (respected and appreciated) help from the tech at CPL, I seem to have made good choices. With all the parts that transferred, it amounts to about a AUD $5,000 build. That's not chicken feed.
 
A clearer view. I'd check this against the memory footprints reported in Task Manager but, assuming Adobe's hogging and there's not a way to enforce more sane behavior via memory limits in settings or workflow shifts, it looks like moving 64+ GB is indeed desirable (personally I'd change software, but I ditched Adobe close to a decade ago so that's not really a fair comparison).
64dc5e6f-2648-4568-96cd-20870af05482.png


Most of my high pixel count work is stacks of 300+ 8.3 MP images, so working sets >2.5 GP (gigapixels). Both Helicon and Zerene page those in and out of physical memory without much trouble, even on my old dual core with 12 GB DDR3 and a SATA III SSD. Zerene does get a bit laggy as it's not nearly as optimized as Helicon—the difference between a one person company and basically a five person company—but that's not a constraint applicable to Adobe.

Perhaps this gives some credence to my previously expressed views on making "a balanced build"?
Two questions appear relevant:
  • Why fairly vague claims of DDR5 problems from a single source aren't being replicated by others. This iteration's a bit clearer on interpretive assumptions but data remains absent.
  • Why an AIC which normally exhibits ~40 °C ΔT around 90% load is remaining at idle temperatures.
These seem more appropriately taken up in your build thread than here but, if it helps, common explanations for the latter are the card's actually unloaded or there's a monitoring error.
 
A clearer view. I'd check this against the memory footprints reported in Task Manager but, assuming Adobe's hogging and there's not a way to enforce more sane behavior via memory limits in settings or workflow shifts, it looks like moving 64+ GB is indeed desirable (personally I'd change software, but I ditched Adobe close to a decade ago so that's not really a fair comparison).
View attachment 344859

Most of my high pixel count work is stacks of 300+ 8.3 MP images, so working sets >2.5 GP (gigapixels). Both Helicon and Zerene page those in and out of physical memory without much trouble, even on my old dual core with 12 GB DDR3 and a SATA III SSD. Zerene does get a bit laggy as it's not nearly as optimized as Helicon—the difference between a one person company and basically a five person company—but that's not a constraint applicable to Adobe.


Two questions appear relevant:
  • Why fairly vague claims of DDR5 problems from a single source aren't being replicated by others. This iteration's a bit clearer on interpretive assumptions but data remains absent.
  • Why an AIC which normally exhibits ~40 °C ΔT around 90% load is remaining at idle temperatures.
These seem more appropriately taken up in your build thread than here but, if it helps, common explanations for the latter are the card's actually unloaded or there's a monitoring error.
Apart from you bandying about terms that have no specifically defined meanings (e.g. "AIC" is not a commonly used acronym or defined abbreviation), I fail to see what you have contributed to this discussion, or Ovi's computer problem.

Further, the same applies to your posts in my own PC build thread. You suggested on several occasions there that what I had specified was not fit for purpose, yet it patently is perfectly suited to its primary function, while remaining capable of considerable enhancement in the future should it become unfit for purpose. After I have established that in my now functioning PC, you seek to take issue with its monitoring software/hardware. Perhaps I just understand how to build PCs that are fit for purpose?

Of course I understand that I could have specified/built an even faster PC than I have, but the costs were prohibitive for minimal potential improvements in performance. Like double the price for a JND (just noticeable difference) in performance.

So you don't like Adobe. You are hardly unique in that, myself included. However, it is the industry standard, for all its faults. I have also been using it for about 17-18 years. I am highly unlikely to live long enough to gain that level of familiarity with any other product, so rather pointless changing at my stage of life.

You constantly push AMD products. You wouldn't be a 'brand ambassador' for them by any chance? In order to equal or exceed the performance of what I have specified using brands I know and trust after some 30+ years of experience with them, I would be doubling the cost of my build, for no perceptible benefit.

Perhaps you could explain yourself more clearly?
 
Back
Top